Can't Display Version of a Python Binary PEX with Click Entry Points

Question

Python Click CLI Application

When you use a Click library to build a Python CLI Application you can do this:

@click.version_option()
def cli():
    '''
    Main Entry Point to Click Interface
    '''

to be able to do this:

[user@host]$ clickapp --version

Click Packaged in pex

But when I package it as a pex file, every other argument, option, commands, sub-command of my click application works, except --version.

When I run (clickapp is now a binary executable pex file):

[user@host]$ ./clickapp --version

I get the following error:

Traceback (most recent call last):
  File "/~/clickapp/.bootstrap/pex/pex.py", line 446, in execute
  File "/~/clickapp/.bootstrap/pex/pex.py", line 378, in _wrap_coverage
  File "/~/clickapp/.bootstrap/pex/pex.py", line 409, in _wrap_profiling
  File "/~/clickapp/.bootstrap/pex/pex.py", line 508, in _execute
  File "/~/clickapp/.bootstrap/pex/pex.py", line 610, in execute_entry
  File "/~/clickapp/.bootstrap/pex/pex.py", line 626, in execute_pkg_resources
  File "/~/.pex/installed_wheels/7bcb1b5bf49ca3f89c348c338d33c04e3d59dfc1/click-7.1.2-py2.py3-none-any.whl/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/~/.pex/installed_wheels/7bcb1b5bf49ca3f89c348c338d33c04e3d59dfc1/click-7.1.2-py2.py3-none-any.whl/click/core.py", line 781, in main
    with self.make_context(prog_name, args, **extra) as ctx:
  File "/~/.pex/installed_wheels/7bcb1b5bf49ca3f89c348c338d33c04e3d59dfc1/click-7.1.2-py2.py3-none-any.whl/click/core.py", line 700, in make_context
    self.parse_args(ctx, args)
  File "/~/.pex/installed_wheels/7bcb1b5bf49ca3f89c348c338d33c04e3d59dfc1/click-7.1.2-py2.py3-none-any.whl/click/core.py", line 1212, in parse_args
    rest = Command.parse_args(self, ctx, args)
  File "/~/.pex/installed_wheels/7bcb1b5bf49ca3f89c348c338d33c04e3d59dfc1/click-7.1.2-py2.py3-none-any.whl/click/core.py", line 1048, in parse_args
    value, args = param.handle_parse_result(ctx, opts, args)
  File "/~/.pex/installed_wheels/7bcb1b5bf49ca3f89c348c338d33c04e3d59dfc1/click-7.1.2-py2.py3-none-any.whl/click/core.py", line 1630, in handle_parse_result
    value = invoke_param_callback(self.callback, ctx, self, value)
  File "/~/.pex/installed_wheels/7bcb1b5bf49ca3f89c348c338d33c04e3d59dfc1/click-7.1.2-py2.py3-none-any.whl/click/core.py", line 123, in invoke_param_callback
    return callback(ctx, param, value)
  File "/~/.pex/installed_wheels/7bcb1b5bf49ca3f89c348c338d33c04e3d59dfc1/click-7.1.2-py2.py3-none-any.whl/click/decorators.py", line 295, in callback
    raise RuntimeError("Could not determine version")
RuntimeError: Could not determine version

Details

The setup.py file:

from setuptools import setup, find_namespace_packages

setup(
    name='clickapp',
    version='0.1.3',
    author='Hamza M Zubair',
    packages=find_namespace_packages(),
    include_package_data=True,
    package_data={
        '': ['*.yaml'],
    },
    classifiers=[
        'Programming Language :: Python :: 3',
        'Operating System :: OS Independent',
        'Natural Language :: English',
        'License :: Other/Proprietary License',
    ],
    python_requires='>=3.6',
    install_requires=[
        'click',
        'pandas',
        'sqlalchemy',
        'jinjasql',
        'pyyaml',
        'joblib',
        'python-dateutil',
        'loguru',
        'pymysql',
        'xgboost',
        'sklearn',
        'wheel',
        'importlib-resources'
    ],
    entry_points='''
        [console_scripts]
        clickapp=clickapp.cli:cli
    ''',
)

The command used to create the pex file:

[user@host]$ python setup.py bdist_pex --bdist-all

Tool Specs

I am building and running the pex file in different systems, using the following versions of libraries/packages. The target machine only has Python and no libraries, because pex file does not require libraries/virtualenv etc.

Build Machine OS: CentOS Linux release 7.8.2003 (Core)  
Build Machine Python: 3.6.8  
setuptools: 51.0.0  
pex: 2.1.21  
click: 7.1.2  
Target Machine OS: CentOS Linus release 7.4.1708 (Core)
Target Machine Python: 3.6.8

What I tried

1. I have tested the full functionality of my clickapp, and every other argument and command works perfectly.

Even this displays the help of my clickapp correctly.

[user@host]$ ./clickapp --help

1. I tried re-building the package several times
2. I have only tested this in Python3.6. I haven't tried different python versions, its slightly difficult to set it up in both the source and target systems.
3. When I removed @click.version_option() I get the error: --version not implemented, which is just as expected
4. I am yet to test it on a 2nd target system, in case some idiosyncrasies of my current target server is causing the error

More Info

What other information should I provide, for helping SO users?

Answer 1

Short answer: setuptools is missing.

---

It looks like you have click v7.1.2. In that version one of the code paths to figure out the version number automatically uses pkg_resources which is a top-level package of setuptools:

                try:
                    import pkg_resources
                except ImportError:
                    pass
                else:
                    for dist in pkg_resources.working_set:
                        scripts = dist.get_entry_map().get("console_scripts") or {}
                        for _, entry_point in iteritems(scripts):
                            if entry_point.module_name == module:
                                ver = dist.version
                                break

-- https://github.com/pallets/click/blob/7.1.2/src/click/decorators.py#L283-L293

So in a way, click depends on setuptools. But there are other code paths that do not require pkg_resources, for example the version number can be set explicitly in the parameters of the decorator (if I am not mistaken): @click.version_option(version='1.2.3') (doc), in such a case setuptools is not a mandatory dependency.

In most cases, it just works because (out of what I would call a coincidence) setuptools is almost always pre-installed (in virtual environments for example). But "almost always" is not the same as "always", and for example it is absolutely possible to have environments (virtual or not) that do not have a setuptools installation. And this looks like it is exactly what happens in the case of a pex-packaged application. I believe pex creates clean virtual environments (i.e. without pip, setuptools or wheel or anything else).

Since setuptools is not declared as a dependency it is not installed, importing pkg_resources fails, and finding the version string fails.

I would say that this is a failure on click's side. They should either declare setuptools as a mandatory dependency, or at least as an optional dependency (in an extra) and document this properly.

A possible fix is to add setuptools as a direct dependency of your application: add setuptools to the list of install_requires in your setup.py.

Note that as far as I can tell, things will change in click v8. Finding out the version string will rely on importlib.metadata from the standard library (back-ported to Python < 3.8 as importlib_metadata) instead of pkg_resources:

• https://github.com/pallets/click/issues/1488
• https://github.com/pallets/click/pull/1582

From the looks of it, it will cause problems again, probably even more. Since click still does not declare importlib-metadata as a dependency for older Python versions where importlib.metadata is not in the standard library (< 3.8), the version string lookup will fail.