Unable to install PyYaml using Poetry

Question

I'm trying to build a virtual environment using Poetry v1.7.1 on Mac, thru poetry install with a pyproject.toml like this:

[tool.poetry]
...

[tool.poetry.dependencies]
python = "^3.9"
mlrun = "1.4.1"
torch = "1.13.1"
transformers = "4.20"
sentence-transformers = "2.2.2"
scikit-learn = "1.2.2"
tensorboard = "2.12.2"
tqdm = "4.65.0"
attr = "0.3.2"
attrs = "23.1.0"
cattrs = "23.2.3"
ruamel-yaml-clib = "0.2.2"
ruamel-yaml = "0.16.10"
emoji = "2.4.0"
s3fs = "2023.1.0"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

However, I always get this error:

Package operations: 38 installs, 2 updates, 0 removals

  • Downgrading attrs (23.2.0 -> 23.1.0)
  • Updating pyyaml (5.3 -> 5.4.1): Failed

  ChefBuildError

  Backend subprocess exited when trying to invoke get_requires_for_build_wheel
  
  running egg_info
  writing lib3/PyYAML.egg-info/PKG-INFO
  writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
  writing top-level names to lib3/PyYAML.egg-info/top_level.txt
  Traceback (most recent call last):
    File "/Users/george/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/Users/george/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/Users/george/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/var/folders/ly/152vxfld44sd8wg4pvj_g6w00000gn/T/tmpyejafm7e/.venv/lib/python3.10/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=['wheel'])
    File "/var/folders/ly/152vxfld44sd8wg4pvj_g6w00000gn/T/tmpyejafm7e/.venv/lib/python3.10/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
      self.run_setup()
    File "/var/folders/ly/152vxfld44sd8wg4pvj_g6w00000gn/T/tmpyejafm7e/.venv/lib/python3.10/site-packages/setuptools/build_meta.py", line 311, in run_setup
      exec(code, locals())
    File "<string>", line 271, in <module>
    File "/var/folders/ly/152vxfld44sd8wg4pvj_g6w00000gn/T/tmpyejafm7e/.venv/lib/python3.10/site-packages/setuptools/__init__.py", line 103, in setup
      return distutils.core.setup(**attrs)
    File "/var/folders/ly/152vxfld44sd8wg4pvj_g6w00000gn/T/tmpyejafm7e/.venv/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
      return run_commands(dist)
    File "/var/folders/ly/152vxfld44sd8wg4pvj_g6w00000gn/T/tmpyejafm7e/.venv/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
      dist.run_commands()
    File "/var/folders/ly/152vxfld44sd8wg4pvj_g6w00000gn/T/tmpyejafm7e/.venv/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
      self.run_command(cmd)
    File "/var/folders/ly/152vxfld44sd8wg4pvj_g6w00000gn/T/tmpyejafm7e/.venv/lib/python3.10/site-packages/setuptools/dist.py", line 963, in run_command
      super().run_command(command)
    File "/var/folders/ly/152vxfld44sd8wg4pvj_g6w00000gn/T/tmpyejafm7e/.venv/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/var/folders/ly/152vxfld44sd8wg4pvj_g6w00000gn/T/tmpyejafm7e/.venv/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 321, in run
      self.find_sources()
    File "/var/folders/ly/152vxfld44sd8wg4pvj_g6w00000gn/T/tmpyejafm7e/.venv/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 329, in find_sources
      mm.run()
    File "/var/folders/ly/152vxfld44sd8wg4pvj_g6w00000gn/T/tmpyejafm7e/.venv/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 551, in run
      self.add_defaults()
    File "/var/folders/ly/152vxfld44sd8wg4pvj_g6w00000gn/T/tmpyejafm7e/.venv/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 589, in add_defaults
      sdist.add_defaults(self)
    File "/var/folders/ly/152vxfld44sd8wg4pvj_g6w00000gn/T/tmpyejafm7e/.venv/lib/python3.10/site-packages/setuptools/command/sdist.py", line 112, in add_defaults
      super().add_defaults()
    File "/var/folders/ly/152vxfld44sd8wg4pvj_g6w00000gn/T/tmpyejafm7e/.venv/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
      self._add_defaults_ext()
    File "/var/folders/ly/152vxfld44sd8wg4pvj_g6w00000gn/T/tmpyejafm7e/.venv/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
      self.filelist.extend(build_ext.get_source_files())
    File "<string>", line 201, in get_source_files
    File "/var/folders/ly/152vxfld44sd8wg4pvj_g6w00000gn/T/tmpyejafm7e/.venv/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
      raise AttributeError(attr)
  AttributeError: cython_sources
  

  at ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/chef.py:164 in _prepare
      160│ 
      161│                 error = ChefBuildError("\n\n".join(message_parts))
      162│ 
      163│             if error is not None:
    → 164│                 raise error from None
      165│ 
      166│             return path
      167│ 
      168│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with pyyaml (5.4.1) not supporting PEP 517 builds. You can verify this by running 'pip wheel --no-cache-dir --use-pep517 "pyyaml (==5.4.1)"'.

Following this thread, I tried adding this line to the toml file, but got an error:

The Poetry configuration is invalid:
  - data.dependencies.PyYAML must be valid exactly by one definition (0 matches found)

I also tried upgrading PIP, and testing different versions by adding pyyaml = "6.0.1" in the toml file...I even tested different Python versions, and installed package one by one. Still no luck.

I also wanted to try this way: https://github.com/yaml/pyyaml/issues/736, but it seems there's no way to set environment variables in pyproject.toml....Can someone please help? Thanks

Answer 1

It turns out you can download the wheel file and place it in the working dir and specify the file name in the toml file:

pyyaml = {file = "PyYAML-5.4.1-cp39-cp39-macosx_10_9_x86_64.whl"}

This now allows me to install PyYaml successfully.

Answer 2

Specifying a wheel file directly in the pyproject.toml, is indeed a practical workaround for the issue you encountered with PyYAML. It bypasses the problem of building the package from source, which was causing the installation to fail.

But:

• When you specify a wheel file directly, Poetry will not resolve the dependencies for PyYAML automatically. Make sure any dependencies PyYAML might have are also correctly managed in your pyproject.toml.
• That ties your project to a specific build of PyYAML, which might affect the portability and reproducibility of your environment on different machines or setups. It is usually a good practice to keep the dependencies as abstract as possible to make sure your project can be set up in diverse environments.
• If you plan to distribute or share your project, including a wheel file directly in the pyproject.toml can make the setup process easier for users on the same platform but can complicate it for users on different platforms or Python versions.

Consider hosting the wheel file in a more accessible location, like a package repository, if you are working in a team or sharing your project with others. That way, you will not need to include the actual wheel file in your project repository. So twine upload dist/*.whl.

When a wheel file is hosted in a package repository, tools like Poetry can more effectively manage dependencies. Poetry can resolve and install dependencies listed in the wheel's metadata, which is not possible when the wheel file is included directly in the project.

Hosting the wheel in a repository makes it accessible regardless of the user's environment. Users do not need to worry about platform-specific wheel files; they can simply point to the repository, and the appropriate wheel file will be used based on their platform and Python version.

By using a package repository, you can easily update or replace the wheel file as new versions of PyYAML are released. Users of your project will automatically fetch the latest version you have specified, without needing to update their local copies of the wheel file.
That centralized approach simplifies the process of keeping dependencies up-to-date and makes sure all users are working with the same version.

It also makes sure collaborators or users of your project can easily set up their environments without manual intervention, as the wheel file is fetched from the repository during the dependency installation process.