Building python pypi wheels, the usual nightmare

Question

I'm trying to build a package and upload it to pypi, I get past that point and the upload is successful, let me walk you through what I'm doing:

setup.py:

from setuptools import setup, find_packages

setup(
    name='project_name',
    version='1.0',
    packages=find_packages(),
    url='url',
    license='license',
    author='author',
    author_email='email_here@some_mail.com',
    description='description',
    install_requires=[
        'oauth2client',
        'pyarrow',
        'pandas',
        'requests',
        'gcloud'
    ],
)

I do:

% python3 setup.py sdist bdist_wheel

followed by

% python3 -m twine upload -u username -p password --repository-url https://test.pypi.org/legacy/ dist/*

both run perfectly fine, no errors / warnings ...

Then I get a url which contains:

% pip install -i https://test.pypi.org/simple/ project_name==1.0

So I create a virtualenv environment and try to install:

% virtualenv test_env
% source test_env/bin/activate
% pip install -i https://test.pypi.org/simple/ project_name==1.0

For some reason I get this at first:

ERROR: Could not find a version that satisfies the requirement gcloud (from project_name==1.0) (from versions: none)
ERROR: No matching distribution found for gcloud (from project_name==1.0)

Then I get something else after retrying (no changes applied) to run the last command and I get other results as well. So ... what's wrong?

simple/ project_name==1.0
Looking in indexes: https://test.pypi.org/simple/
Collecting project_name==1.0
  Downloading https://test-files.pythonhosted.org/packages/08/38/b040820ceddc63a87596a5a29ce3a5d1b309555238d7ae063835e8c8ea8a/project_name-1.0-py3-none-any.whl (9.1 kB)
Collecting pyarrow
  Downloading https://test-files.pythonhosted.org/packages/c1/82/04249512513b31d7cd9f6fa63cf0d1c64c4705da32e3c3ece9d676e235ff/pyarrow-1.0.1.tar.gz (1.3 MB)
     |████████████████████████████████| 1.3 MB 594 kB/s 
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /Users/username/Desktop/testenv/bin/python /Users/username/Desktop/testenv/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/rp/xxjnjsvn70g2ndh68l0g10bh0000gn/T/pip-build-env-cnc0ds0l/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://test.pypi.org/simple/ -- 'cython >= 0.29' 'numpy==1.14.5; python_version<'"'"'3.7'"'"'' 'numpy==1.16.0; python_version>='"'"'3.7'"'"'' setuptools setuptools_scm wheel
       cwd: None
  Complete output (4 lines):
  Looking in indexes: https://test.pypi.org/simple/
  Ignoring numpy: markers 'python_version < "3.7"' don't match your environment
  ERROR: Could not find a version that satisfies the requirement cython>=0.29 (from versions: 0.23.4)
  ERROR: No matching distribution found for cython>=0.29
  ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/username/Desktop/testenv/bin/python /Users/username/Desktop/testenv/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/rp/xxjnjsvn70g2ndh68l0g10bh0000gn/T/pip-build-env-cnc0ds0l/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://test.pypi.org/simple/ -- 'cython >= 0.29' 'numpy==1.14.5; python_version<'"'"'3.7'"'"'' 'numpy==1.16.0; python_version>='"'"'3.7'"'"'' setuptools setuptools_scm wheel Check the logs for full command output.

Note: The package is fine, I'm able to install it on my system interpreter (python3.8) of my mbp but that is because all the dependencies numpy, gcloud ... are already installed.

Answer 1

I think this is because pip is looking for the package dependancies in https://test.pypi.org/simple/ where they don't exist.

Try:

pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple your-package

This will pull your package from test.pypi but then fall back on regular pypi when pip fails to find the dependancies there.