Downgrade setuptools inside tox dependencies

Question

I have a gdal dependency in my tests.

I use rocky-linux 8. epel 8 provides gdal 3.0.4, so I must install the same version in python, but this version is incompatible with the latest version of setuptools. (See: Error building pygdal: Unknown distribution option: 'use_2to3_fixers' and 'use_2to3_exclude_fixers')

Installing gdal 3.0.4 works fine with setuptools 57.5.0. In a virtual environment, from the command line:

python3.9 -m venv ~/venv
source ~/venv/bin/bin/activate
pip install setuptools==57.5.0
pip install gdal==3.0.4

But it doesn't work when I try to build the environment with tox. I have tried in my tox.ini:

[tox]
env_list = py39

[testenv]
deps =
    setuptools==57.5.0
    gdal==3.0.4
    pytest
commands =
    python -m pytest

How can I build an environment containing gdal with tox on a distribution like redhat 8 ?

My package is a library dealing numpy matrices with pybind11. gdal is used to load data, it's not a dependency of my package.

Running tox -rvv:

py39: 173 I find interpreter for spec PythonSpec(major=3, minor=9) [virtualenv/discovery/builtin.py:56]
py39: 173 I proposed PythonInfo(spec=CPython3.9.16.final.0-64, exe=/opt/python/bin/python3.9, platform=linux, version='3.9.16 (main, Mar 28 2023, 07:59:14) \n[GCC 8.5.0 20210514 (Red Hat 8.5.0-16)]', encoding_fs_io=utf-8-utf-8) [virtualenv/discovery/builtin.py:63]
py39: 173 D accepted PythonInfo(spec=CPython3.9.16.final.0-64, exe=/opt/python/bin/python3.9, platform=linux, version='3.9.16 (main, Mar 28 2023, 07:59:14) \n[GCC 8.5.0 20210514 (Red Hat 8.5.0-16)]', encoding_fs_io=utf-8-utf-8) [virtualenv/discovery/builtin.py:65]
py39: 174 D filesystem is case-sensitive [virtualenv/info.py:24]
py39: 194 I create virtual environment via CPython3Posix(dest=/home/usertest/test/.tox/py39, clear=False, no_vcs_ignore=False, global=False) [virtualenv/run/session.py:48]
py39: 194 D create folder /home/usertest/test/.tox/py39/bin [virtualenv/util/path/_sync.py:9]
py39: 194 D create folder /home/usertest/test/.tox/py39/lib/python3.9/site-packages [virtualenv/util/path/_sync.py:9]
py39: 194 D write /home/usertest/test/.tox/py39/pyvenv.cfg [virtualenv/create/pyenv_cfg.py:30]
py39: 194 D     home = /opt/python/bin [virtualenv/create/pyenv_cfg.py:34]
py39: 194 D     implementation = CPython [virtualenv/create/pyenv_cfg.py:34]
py39: 194 D     version_info = 3.9.16.final.0 [virtualenv/create/pyenv_cfg.py:34]
py39: 194 D     virtualenv = 20.21.0 [virtualenv/create/pyenv_cfg.py:34]
py39: 194 D     include-system-site-packages = false [virtualenv/create/pyenv_cfg.py:34]
py39: 195 D     base-prefix = /opt/python [virtualenv/create/pyenv_cfg.py:34]
py39: 195 D     base-exec-prefix = /opt/python [virtualenv/create/pyenv_cfg.py:34]
py39: 195 D     base-executable = /opt/python/bin/python3.9 [virtualenv/create/pyenv_cfg.py:34]
py39: 195 D symlink /opt/python/bin/python3.9 to /home/usertest/test/.tox/py39/bin/python [virtualenv/util/path/_sync.py:28]
py39: 195 D create virtualenv import hook file /home/usertest/test/.tox/py39/lib/python3.9/site-packages/_virtualenv.pth [virtualenv/create/via_global_ref/api.py:89]
py39: 195 D create /home/usertest/test/.tox/py39/lib/python3.9/site-packages/_virtualenv.py [virtualenv/create/via_global_ref/api.py:92]
py39: 195 D ============================== target debug ============================== [virtualenv/run/session.py:50]
py39: 196 D debug via /home/usertest/test/.tox/py39/bin/python /opt/python/lib/python3.9/site-packages/virtualenv/create/debug.py [virtualenv/create/creator.py:193]
py39: 195 D {
  "sys": {
    "executable": "/home/usertest/test/.tox/py39/bin/python",
    "_base_executable": "/home/usertest/test/.tox/py39/bin/python",
    "prefix": "/home/usertest/test/.tox/py39",
    "base_prefix": "/opt/python",
    "real_prefix": null,
    "exec_prefix": "/home/usertest/test/.tox/py39",
    "base_exec_prefix": "/opt/python",
    "path": [
      "/opt/python/lib/python39.zip",
      "/opt/python/lib/python3.9",
      "/opt/python/lib/python3.9/lib-dynload",
      "/home/usertest/test/.tox/py39/lib/python3.9/site-packages"
    ],
    "meta_path": [
      "<class '_virtualenv._Finder'>",
      "<class '_frozen_importlib.BuiltinImporter'>",
      "<class '_frozen_importlib.FrozenImporter'>",
      "<class '_frozen_importlib_external.PathFinder'>"
    ],
    "fs_encoding": "utf-8",
    "io_encoding": "utf-8"
  },
  "version": "3.9.16 (main, Mar 28 2023, 07:59:14) \n[GCC 8.5.0 20210514 (Red Hat 8.5.0-16)]",
  "makefile_filename": "/opt/python/lib/python3.9/config-3.9-x86_64-linux-gnu/Makefile",
  "os": "<module 'os' from '/opt/python/lib/python3.9/os.py'>",
  "site": "<module 'site' from '/opt/python/lib/python3.9/site.py'>",
  "datetime": "<module 'datetime' from '/opt/python/lib/python3.9/datetime.py'>",
  "math": "<module 'math' from '/opt/python/lib/python3.9/lib-dynload/math.cpython-39-x86_64-linux-gnu.so'>",
  "json": "<module 'json' from '/opt/python/lib/python3.9/json/__init__.py'>"
} [virtualenv/run/session.py:51]
py39: 217 I add seed packages via FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/usertest/.local/share/virtualenv) [virtualenv/run/session.py:55]
py39: 219 D install pip from wheel /opt/python/lib/python3.9/site-packages/virtualenv/seed/wheels/embed/pip-23.0.1-py3-none-any.whl via CopyPipInstall [virtualenv/seed/embed/via_app_data/via_app_data.py:47]
py39: 219 D install wheel from wheel /opt/python/lib/python3.9/site-packages/virtualenv/seed/wheels/embed/wheel-0.38.4-py3-none-any.whl via CopyPipInstall [virtualenv/seed/embed/via_app_data/via_app_data.py:47]
py39: 219 D install setuptools from wheel /opt/python/lib/python3.9/site-packages/virtualenv/seed/wheels/embed/setuptools-67.4.0-py3-none-any.whl via CopyPipInstall [virtualenv/seed/embed/via_app_data/via_app_data.py:47]
py39: 220 D copy directory /home/usertest/.local/share/virtualenv/wheel/3.9/image/1/CopyPipInstall/pip-23.0.1-py3-none-any/pip to /home/usertest/test/.tox/py39/lib/python3.9/site-packages/pip [virtualenv/util/path/_sync.py:36]
py39: 220 D copy directory /home/usertest/.local/share/virtualenv/wheel/3.9/image/1/CopyPipInstall/wheel-0.38.4-py3-none-any/wheel to /home/usertest/test/.tox/py39/lib/python3.9/site-packages/wheel [virtualenv/util/path/_sync.py:36]
py39: 220 D copy /home/usertest/.local/share/virtualenv/wheel/3.9/image/1/CopyPipInstall/setuptools-67.4.0-py3-none-any/distutils-precedence.pth to /home/usertest/test/.tox/py39/lib/python3.9/site-packages/distutils-precedence.pth [virtualenv/util/path/_sync.py:36]
py39: 221 D copy directory /home/usertest/.local/share/virtualenv/wheel/3.9/image/1/CopyPipInstall/setuptools-67.4.0-py3-none-any/_distutils_hack to /home/usertest/test/.tox/py39/lib/python3.9/site-packages/_distutils_hack [virtualenv/util/path/_sync.py:36]
py39: 221 D copy directory /home/usertest/.local/share/virtualenv/wheel/3.9/image/1/CopyPipInstall/setuptools-67.4.0-py3-none-any/pkg_resources to /home/usertest/test/.tox/py39/lib/python3.9/site-packages/pkg_resources [virtualenv/util/path/_sync.py:36]
py39: 225 D copy directory /home/usertest/.local/share/virtualenv/wheel/3.9/image/1/CopyPipInstall/wheel-0.38.4-py3-none-any/wheel-0.38.4.dist-info to /home/usertest/test/.tox/py39/lib/python3.9/site-packages/wheel-0.38.4.dist-info [virtualenv/util/path/_sync.py:36]
py39: 226 D copy /home/usertest/.local/share/virtualenv/wheel/3.9/image/1/CopyPipInstall/wheel-0.38.4-py3-none-any/wheel-0.38.4.virtualenv to /home/usertest/test/.tox/py39/lib/python3.9/site-packages/wheel-0.38.4.virtualenv [virtualenv/util/path/_sync.py:36]
py39: 228 D generated console scripts wheel3.9 wheel3 wheel-3.9 wheel [virtualenv/seed/embed/via_app_data/pip_install/base.py:41]
py39: 231 D copy directory /home/usertest/.local/share/virtualenv/wheel/3.9/image/1/CopyPipInstall/setuptools-67.4.0-py3-none-any/setuptools to /home/usertest/test/.tox/py39/lib/python3.9/site-packages/setuptools [virtualenv/util/path/_sync.py:36]
py39: 257 D copy directory /home/usertest/.local/share/virtualenv/wheel/3.9/image/1/CopyPipInstall/setuptools-67.4.0-py3-none-any/setuptools-67.4.0.dist-info to /home/usertest/test/.tox/py39/lib/python3.9/site-packages/setuptools-67.4.0.dist-info [virtualenv/util/path/_sync.py:36]
py39: 258 D copy /home/usertest/.local/share/virtualenv/wheel/3.9/image/1/CopyPipInstall/setuptools-67.4.0-py3-none-any/setuptools-67.4.0.virtualenv to /home/usertest/test/.tox/py39/lib/python3.9/site-packages/setuptools-67.4.0.virtualenv [virtualenv/util/path/_sync.py:36]
py39: 259 D generated console scripts  [virtualenv/seed/embed/via_app_data/pip_install/base.py:41]
py39: 284 D copy directory /home/usertest/.local/share/virtualenv/wheel/3.9/image/1/CopyPipInstall/pip-23.0.1-py3-none-any/pip-23.0.1.dist-info to /home/usertest/test/.tox/py39/lib/python3.9/site-packages/pip-23.0.1.dist-info [virtualenv/util/path/_sync.py:36]
py39: 284 D copy /home/usertest/.local/share/virtualenv/wheel/3.9/image/1/CopyPipInstall/pip-23.0.1-py3-none-any/pip-23.0.1.virtualenv to /home/usertest/test/.tox/py39/lib/python3.9/site-packages/pip-23.0.1.virtualenv [virtualenv/util/path/_sync.py:36]
py39: 285 D generated console scripts pip-3.9 pip pip3.9 pip3 [virtualenv/seed/embed/via_app_data/pip_install/base.py:41]
py39: 285 I add activators for Bash, CShell, Fish, Nushell, PowerShell, Python [virtualenv/run/session.py:61]
py39: 286 D write /home/usertest/test/.tox/py39/pyvenv.cfg [virtualenv/create/pyenv_cfg.py:30]
py39: 286 D     home = /opt/python/bin [virtualenv/create/pyenv_cfg.py:34]
py39: 286 D     implementation = CPython [virtualenv/create/pyenv_cfg.py:34]
py39: 286 D     version_info = 3.9.16.final.0 [virtualenv/create/pyenv_cfg.py:34]
py39: 286 D     virtualenv = 20.21.0 [virtualenv/create/pyenv_cfg.py:34]
py39: 286 D     include-system-site-packages = false [virtualenv/create/pyenv_cfg.py:34]
py39: 286 D     base-prefix = /opt/python [virtualenv/create/pyenv_cfg.py:34]
py39: 286 D     base-exec-prefix = /opt/python [virtualenv/create/pyenv_cfg.py:34]
py39: 287 D     base-executable = /opt/python/bin/python3.9 [virtualenv/create/pyenv_cfg.py:34]
py39: 289 W install_deps> python -I -m pip install gdal==3.0.4 pytest setuptools==57.5.0 [tox/tox_env/api.py:428]
Looking in indexes: https://pypi.org/simple, http://my-nexus.docker:8081/repository/pypi-synpkgs/simple
Collecting gdal==3.0.4
  Using cached GDAL-3.0.4.tar.gz (577 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]
      /home/usertest/test/.tox/py39/lib/python3.9/site-packages/setuptools/_distutils/dist.py:265: UserWarning: Unknown distribution option: 'use_2to3_fixers'
        warnings.warn(msg)
      /home/usertest/test/.tox/py39/lib/python3.9/site-packages/setuptools/_distutils/dist.py:265: UserWarning: Unknown distribution option: 'use_2to3_exclude_fixers'
        warnings.warn(msg)
      error in GDAL setup command: use_2to3 is invalid.
      WARNING: numpy not available!  Array support will not be enabled
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
py39: 1491 C exit 1 (1.20 seconds) /home/usertest/test> python -I -m pip install gdal==3.0.4 pytest setuptools==57.5.0 pid=26032 [tox/execute/api.py:275]
  py39: FAIL code 1 (1.32 seconds)
  evaluation failed :( (1.41 seconds)

gdal setup.py can be found here.

Answer 1

Inspired by this, gdal can be installed as a second step in the commands section of the testenv, after installing the correct version of setuptools.

[tox]
env_list = py39

[testenv]
deps =
    setuptools==57.5.0
    pytest
commands =
    pip install gdal==3.0.4
    python -m pytest