How to access a module installed by setuptools' install_requires within setup.py?

Question

I'm writing a setup.py to install my package reboundx, which has a single dependency, rebound. My package builds an extension libreboundx.so that needs to link to librebound.so in setup.py

rebxExt = Extension('libreboundx', libraries=['rebound'], library_dirs = [rebound_path]...)

I'd like to be able to use install_requires in the setup(...) call to build the reboundx module to make sure the right version of rebound is installed. Is there any way to resolve the circularity?

If rebound is not installed, I would somehow need setuptools to detect this through install_requires, install rebound, and THEN find the right paths and build the extension libreboundx.so.

Answer 1

You should use the setup_requires argument to setup(). From the docs,

setup_requires

A string or list of strings specifying what other distributions need to be present in order for the setup script to run. setuptools will attempt to obtain these (even going so far as to download them using EasyInstall) before processing the rest of the setup script or commands. This argument is needed if you are using distutils extensions as part of your build process; for example, extensions that process setup() arguments and turn them into EGG-INFO metadata files.

(Note: projects listed in setup_requires will NOT be automatically installed on the system where the setup script is being run. They are simply downloaded to the ./.eggs directory if they’re not locally available already. If you want them to be installed, as well as being available when the setup script is run, you should add them to install_requires and setup_requires.)

https://pythonhosted.org/setuptools/setuptools.html#new-and-changed-setup-keywords

Edit: It should create an egg directory for each entry in setup_requires. However, I just tried this with rebound and it actually fails to build under easy install.

$> python2 setup.py install
install_dir .
warning: no files found matching 'src/rebound.h'
src/rebound.c:38:21: fatal error: rebound.h: No such file or directory
compilation terminated.
Traceback (most recent call last):
  File "setup.py", line 187, in <module>
    url="http://www.mathics.github.io/",   # project home page, if any
  File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
    _setup_distribution = dist = klass(attrs)
  File "/usr/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/setuptools/dist.py", line 221, in __init__
    self.fetch_build_eggs(attrs.pop('setup_requires'))
  File "/usr/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/setuptools/dist.py", line 245, in fetch_build_eggs
    parse_requirements(requires), installer=self.fetch_build_egg
  File "/usr/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/pkg_resources.py", line 544, in resolve
    dist = best[req.key] = env.best_match(req, self, installer)
  File "/usr/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/pkg_resources.py", line 786, in best_match
    return self.obtain(req, installer) # try and download/install
  File "/usr/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/pkg_resources.py", line 798, in obtain
    return installer(requirement)
  File "/usr/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/setuptools/dist.py", line 293, in fetch_build_egg
    return cmd.easy_install(req)
  File "/usr/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/setuptools/command/easy_install.py", line 582, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "/usr/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/setuptools/command/easy_install.py", line 612, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "/usr/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/setuptools/command/easy_install.py", line 802, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/usr/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/setuptools/command/easy_install.py", line 1079, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/usr/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/setuptools/command/easy_install.py", line 1070, in run_setup
    raise DistutilsError("Setup script exited with %s" % (v.args[0],))
distutils.errors.DistutilsError: Setup script exited with error: command 'gcc' failed with exit status 1

Answer 2

(If you like this approach, I can elaborate further) Consider this example. I use ctypes to load a shared library. Then get the version of the library using a function call.

>>> import ctypes
>>> x = ctypes.cdll.LoadLibrary("libc.so.6")
>>> x.gnu_get_libc_version
<_FuncPtr object at 0x7f9a08e3e460>
>>> getversion = x.gnu_get_libc_version
>>> getversion.restype = ctypes.c_char_p
>>> getversion()
'2.19'

You could do something similar with rebound.so. Use a try statement to load librebound. If it is the wrong version or it is not found in the standard search path for libraries, then compile librebound.

#setup.py
import ctypes
try:
    x = ctypes.cdll.LoadLibrary("librebound.so")
    x.restype = ctypes.c_char_p
    version = x.get_version()
    major,minor = version.split(".")
    if int(major) < REQUIRED_VERSION:
        raise False, "LIBREBOUND VERSION %s IS NOT SUPPORTED"%(str(version))

except:
    pass
    #invoke rebound's makefile
    #probably install the new library to lib path.
    #link to new rebound 

rebxExt = Extension('libreboundx', libraries=['rebound'], library_dirs = [rebound_path]...)

Answer 3

It seems setuptools does make this possible through the setup_requires keyword, as pointed out by sn6uv. However, I found it difficult to find documentation and/or examples that showed how to use it. From what I did find, it also seemed like it might be a bit delicate/complicated for the uninitiated.

For one, when using pip, the dependencies in setup_requires are installed differently than everything else (see Install package which has setup_requires from local source distributions). It also seems like if you need to import the dependency within setup.py, you need to subclass the install command in order to delay the import until it's available (deep in a thread on the distutils mailing list arguing over setup_requires, Donald Stufft talks about this and links to an example: https://mail.python.org/pipermail/distutils-sig/2015-March/025882.html).

So I gave up and implemented something similar to the manual check suggested by goCards. Just thought I'd post the links for the more ambitious who come across this.