Continuous Machine Learning pipeline broken by tensorflow requirements installation

Question

I am doing Continuous Machine Learning (https://cml.dev/) on my own GitLab server. My goal is to test the basic Continuous Machine Learning pipeline with python script as an example.

My .gitlab-ci.yml file is a basic one:

stages:
  - cml_run

cml:
  stage: cml_run
  image: dvcorg/cml-py3:latest
  script:
    - pip3 install -r requirements.txt
    - python train.py

    - cat metrics.txt >> report.md
    - cml-publish confusion_matrix.png --md >> report.md
    - cml-send-comment report.md

For pandas,sklearn and Keras in the requirements.txt, there is a successful installation. But I receive a pipeline broken by TensorFlow requirements installation

  $ pip3 install -r requirements.txt
 Collecting pandas
   Downloading pandas-1.0.5-cp36-cp36m-manylinux1_x86_64.whl (10.1 MB)
 Collecting sklearn
   Downloading sklearn-0.0.tar.gz (1.1 kB)
 Collecting keras
   Downloading Keras-2.4.3-py2.py3-none-any.whl (36 kB)
 Collecting tensorflow
   Downloading tensorflow-2.2.0-cp36-cp36m-manylinux2010_x86_64.whl (516.2 MB)
 ERROR: Exception:
 Traceback (most recent call last):
   File "/usr/local/lib/python3.6/dist-packages/pip/_internal/cli/base_command.py", line 188, in _main
     status = self.run(options, args)
   File "/usr/local/lib/python3.6/dist-packages/pip/_internal/cli/req_command.py", line 185, in wrapper
     return func(self, options, args)
   File "/usr/local/lib/python3.6/dist-packages/pip/_internal/commands/install.py", line 333, in run
     reqs, check_supported_wheels=not options.target_dir
   File "/usr/local/lib/python3.6/dist-packages/pip/_internal/resolution/legacy/resolver.py", line 179, in resolve
     discovered_reqs.extend(self._resolve_one(requirement_set, req))
   File "/usr/local/lib/python3.6/dist-packages/pip/_internal/resolution/legacy/resolver.py", line 362, in _resolve_one
     abstract_dist = self._get_abstract_dist_for(req_to_install)
   File "/usr/local/lib/python3.6/dist-packages/pip/_internal/resolution/legacy/resolver.py", line 314, in _get_abstract_dist_for
     abstract_dist = self.preparer.prepare_linked_requirement(req)
   File "/usr/local/lib/python3.6/dist-packages/pip/_internal/operations/prepare.py", line 469, in prepare_linked_requirement
     hashes=hashes,
   File "/usr/local/lib/python3.6/dist-packages/pip/_internal/operations/prepare.py", line 259, in unpack_url
     hashes=hashes,
   File "/usr/local/lib/python3.6/dist-packages/pip/_internal/operations/prepare.py", line 130, in get_http_url
     link, downloader, temp_dir.path, hashes
   File "/usr/local/lib/python3.6/dist-packages/pip/_internal/operations/prepare.py", line 281, in _download_http_url
     for chunk in download.chunks:
   File "/usr/local/lib/python3.6/dist-packages/pip/_internal/cli/progress_bars.py", line 166, in iter
     for x in it:
   File "/usr/local/lib/python3.6/dist-packages/pip/_internal/network/utils.py", line 39, in response_chunks
     decode_content=False,
   File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/urllib3/response.py", line 564, in stream
     data = self.read(amt=amt, decode_content=decode_content)
   File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/urllib3/response.py", line 507, in read
     data = self._fp.read(amt) if not fp_closed else b""
   File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/cachecontrol/filewrapper.py", line 65, in read
     self._close()
   File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/cachecontrol/filewrapper.py", line 52, in _close
     self.__callback(self.__buf.getvalue())
   File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/cachecontrol/controller.py", line 309, in cache_response
     cache_url, self.serializer.dumps(request, response, body=body)
   File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/cachecontrol/serialize.py", line 72, in dumps
     return b",".join([b"cc=4", msgpack.dumps(data, use_bin_type=True)])
   File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/msgpack/__init__.py", line 35, in packb
     return Packer(**kwargs).pack(o)
   File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/msgpack/fallback.py", line 936, in pack
     self._pack(obj)
   File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/msgpack/fallback.py", line 920, in _pack
     len(obj), dict_iteritems(obj), nest_limit - 1
   File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/msgpack/fallback.py", line 1021, in _pack_map_pairs
     self._pack(v, nest_limit - 1)
   File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/msgpack/fallback.py", line 920, in _pack
     len(obj), dict_iteritems(obj), nest_limit - 1
   File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/msgpack/fallback.py", line 1021, in _pack_map_pairs
     self._pack(v, nest_limit - 1)
   File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/msgpack/fallback.py", line 865, in _pack
     return self._buffer.write(obj)
 MemoryError

Any ideas on how to overcome his issue with CML pipeline on GitLab?

Answer 1

It is difficult to diagnose without knowing more about your .gitlab-ci.yml file. But based on the MemoryError message, it seems runner does not have enough memory to install Tensorflow in addition to your other project dependencies.

You can try installing TF with the --no-cache-dir flag