python loading c lib with CDLL, doesn't see libraries in the python path

Question

I'm trying to get some open source academic code working (the project home is here). It is a big C++ codebase with a (very) thin python wrapper which uses CDLL to load the C++ and call some C functions that are available to allow primitive python scripting of the code.

However, the initial import code crashes because it can't find the .so files sitting next to it in site-packages:

in the installed file:

from ctypes import *

try:
  self.lib = CDLL("_lammps.so")
except:
  try:
    self.lib = CDLL("_lammps_serial.so")
  except:
    raise OSError,"Could not load LAMMPS dynamic library"

and in a script or the interpreter:

from lammps import lammps
l = lammps()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "lammps.py", line 42, in __init__
    raise OSError,"Could not load LAMMPS dynamic library"
OSError: Could not load LAMMPS dynamic library

Other answers might seem to have this covered, but this only works if CDLL() is called within the script actually invoked (or the working directory of the prompt that ran the interpreter) - i.e. if the 'relative path' is in user-space, rather than python-library-space.

How do we reliably install for import a C/C++ library that we built ourselves? Short of polluting the system library locations like /usr/lib, which isn't very pythonic, I can't see an easy solution.

(EDIT: corrected function names, unclear refactoring unhelpful! sorry!)

Answer 1

You could use the __file__ variable in the package that is doing the importing. Just use the various functions of os.path to extract the full, absolute directory path from __file__, and then join it to your library filename. Something like:

temp = os.path.abspath(__file__)
temp = os.path.realpath(temp)
temp = os.path.dirname(temp)
temp = os.path.join(temp, "_lammps.so")
lib = CDLL(path)

You might also want to try various variations of your core filename (i.e., with .dll or .dylib instead of .so, and with and without a lib prefix, and perhaps even with version numbers appended) if you want to be platform-independent, and your build system could possibly produce such stuff. You could then either try several versions, or just use glob.glob to help you find an acceptable one.

I've got to say I think it's odd that no such function exists in the standard library. The ctypes.util.find_library isn't quite flexible (or thorough) enough for this kind of use (which I would think was widespread). Even just a function that searched through PYTHONPATH for the file would have been quite useful (though not hard to write).

Then again, it seems that if you just add the right directory to LD_LIBRARY_PATH, you should be able to load it.

Answer 2

Im on linux, all I did to fix this issue was put in the absolute path from the os module, and it works

from ctypes import *
import os

xss = cdll.LoadLibrary(os.path.abspath("libxss.so.1"))
print xss.xss_test_1()

This is python 2.7 as well.

Answer 3

You must specify the absolute path. Try the following:

import os
from ctypes import *

try:
  slib = os.getcwd() + '/' +'_lammps.so'
  hlibc = CDLL(slib)
  hilbc.FunctionName()
except:
  try:
    slib = os.getcwd() + '/' +'_lammps_serial.so'
    hlibc = CDLL(slib)
  except:
    raise OSError,"Could not load LAMMPS dynamic library"

Answer 4

Put the relevant folder into the PATH variable before importing the relevant libs via ctypes.

For example if the libs (dll, ...) are inside the same folder as the .py file:

import os
os.environ["PATH"] += os.pathsep + os.path.dirname(os.path.abspath(__file__))

This will not write anything to your system PATH. It will just set it for this instance.

Answer 5

Run under strace -eopen, you will see something like this:

open("tls/x86_64/_lammps.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("tls/_lammps.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("x86_64/_lammps.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("_lammps.so", O_RDONLY|O_CLOEXEC)  = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 6
open("/lib/_lammps.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/_lammps.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

Which shows you all the locations where python ctypes looks for your library. So far I was unable to find a runtime environment variable tweak to make it to add search locations on my system, perhaps you have to use absolute paths.