Packaging local module with pex

Question

I'm trying to package my local module with pex and I can't seem to manage to.

I've created a simple project:

→ python --version
Python 2.7.10
→ pex --version
pex 1.1.15
→ tree .
.
├── bla
│   ├── __init__.py
│   └── main.py
└── setup.py

bla/__init__.py

import main

bla/main.py

if __name__ == '__main__':
   print 'yo'

Which to me seems like the simplest project possible.

→ pex -v . -o v.pex --disable-cache
Traceback (most recent call last):
  File "/Users/Charly/repos/load_tester/venv/bin/pex", line 11, in <module>
   sys.exit(main())
 File "/Users/Charly/repos/load_tester/venv/lib/python2.7/site-packages/pex/bin/pex.py", line 540, in main
    pex_builder = build_pex(reqs, options, resolver_options_builder)
 File "/Users/Charly/repos/load_tester/venv/lib/python2.7/site-packages/pex/bin/pex.py", line 475, in build_pex
    resolvables = [Resolvable.get(arg, resolver_option_builder) for arg in args]
  File "/Users/Charly/repos/load_tester/venv/lib/python2.7/site-packages/pex/resolvable.py", line 61, in get
    raise cls.InvalidRequirement('Unknown requirement type: %s' % resolvable_string)
 pex.resolvable.InvalidRequirement: Unknown requirement type: .

Also tried to do python setup.py bdist_pex but this failed as well as it can't find the command.

It really seems like I'm misunderstanding something fundamental but I can't figure out what.

Answer 1

I recently had a bit of a fight with pex trying to make it include local modules. What I learned is:

1. You must provide a valid setup.py file for your module(s) in order for this to work, and:
2. You must specify the application's entry point

This was tricky to figure out for several reasons. From reading the documentation, I was able to infer that the correct command in my case should be something like this:

$ pex . -v -e usersnotifier:main -o usersnotifier.pex

However, when I tried this, I kept getting an error saying:

pex.resolvable.InvalidRequirement: Unknown requirement type: .

A web search for this error turns up—as its first hit—this Github issue, which is still open as I type this. (UPDATE: This bug has been fixed as of September 2019.) A spent a long while thinking that the above command wasn't working because of this bug. I attempted to downgrade setuptools and made a half-dozen other fruitless attempts to 'fix' the problem before this SO answer hinted at the necessity of supplying a setup.py file. (That Github issue turned out to be a red herring. The setuptools bug it mentions has since been fixed, from what I can tell.)

So... I wrote a setup.py file. At first, I kept getting that error saying Unknown requirement type: . But then I realized that my setup.py simply contained a dead-obvious typographical error. The error message emitted by pex in this case was actually quite clear, but it was followed by a large-ish stack trace and the Unknown requirement type: . message. I just wasn't paying close attention and missed it for longer than I care to admit.

I finally noticed my typo and fixed it, but another flaw in my setup.py was failing to include my local modules. pex worked in this case, but the generated file didn't:

$ pex . -v -e usersnotifier:main -o usersnotifier.pex --disable-cache                                                                                                                     
  usersnotifier 0.1: Resolving distributions :: Packaging paho-mqtt    
  pyinotify 0.9.6
  paho-mqtt 1.3.1
pex: Building pex: 2704.3ms                                        
pex:   Resolving distributions: 2393.2ms
pex:       Packaging usersnotifier: 319.3ms
pex:       Packaging pyinotify: 347.4ms
pex:       Packaging paho-mqtt: 361.1ms
Saving PEX file to usersnotifier.pex

$ ./usersnotifier.pex 
Traceback (most recent call last):
  File ".bootstrap/_pex/pex.py", line 367, in execute
  File ".bootstrap/_pex/pex.py", line 293, in _wrap_coverage
  File ".bootstrap/_pex/pex.py", line 325, in _wrap_profiling
  File ".bootstrap/_pex/pex.py", line 410, in _execute
  File ".bootstrap/_pex/pex.py", line 468, in execute_entry
  File ".bootstrap/_pex/pex.py", line 482, in execute_pkg_resources
  File ".bootstrap/pkg_resources/__init__.py", line 2297, in resolve
ImportError: No module named 'usersnotifier'

Here's the bare-bones setup.py that finally worked for me:

from setuptools import setup                                                                                                                                                              

setup(
    name='usersnotifier',
    version='0.1',
    py_modules=['usersnotifier', 'userswatcher'],
    install_requires=[
        'paho-mqtt>=1.3.1',
        'pyinotify>=0.9.6',
    ],
    include_package_data=True,
    zip_safe=False
)

The reason it hadn't worked before was that I was accidentally passing the parameter py_module to setup() rather than py_modules (plural). ¯\(ツ)/¯

The final hurdle I encountered was mentioned in @cmcginty's answer to this question, namely: unless your module version number changes, pex will cache/reuse artifacts from the last time you ran it. So, if you fix a problem in your setup.py and re-run pex, it won't actually incorporate your changes unless you: a) bump the version number, or b) pass --disable-cache when invoking pex.

At the end of the day, the whole thing turned into an exercise in writing a proper setup.py, and running:

$ pex . -v -e mymodule:main -o mymodule.pex --disable-cache

Here are a few tips I can offer (possibly to a future version of my self):

TIP 1

Use python setup.py sdist to test your setup.py file. It's surprisingly easy to screw this up, and there's no point involving pex until you're sure your package has the right contents. After running python setup.py sdist, try installing the source package it generates (located in the dist folder) into a fresh venv and see whether it contains all the files you expect. Only move on to invoking pex after this is working.

TIP 2

Always pass --disable-cache to pex unless you have a good reason not to.

TIP 3

While troubleshooting all of these issues, I discovered that I could run:

$ unzip mymodule.pex

to extract the contents of the PEX file. This can be helpful in resolving any discrepancies that remain between your sdist package contents and your pex-ified application.

Answer 2

one way to do is:

1. create create a source distribution (tarball, zip file, etc.) using python setup.py sdist

2. then run pex command with -f DIST_DIR switch

   eg. pex $(pip freeze) -o aflaskapp.pex -e 'aflaskapp.app' -f dist -v

Answer 3

Some additional details I discovered. If you run:

pex .

It will attempt to use the setup.py build in the directory. If you do not have that, then pex will fail.

By default, pex will cache the local package output, so if you change your setup.py then use the command to make sure your the changes are applied on the next pex run.

pex . --disable-cache