Theano cannot import pygpu/cudnn (macOS, CUDA 8)

Question

The problem is that I can't import theano. I know there's several other questions about same issue but none of them have worked for me and usually they are either with older versions or different OS. I've been trying to fix this for a few days now and I've tried every fix and trick I can find so I can't figure out anything else to try anymore than to ask here.

If I try python3 -c "import theano" without sudo, I get the following:

ERROR (theano.gpuarray): Could not initialize pygpu, support disabled
Traceback (most recent call last):
  File "/Users/np/miniconda3/lib/python3.6/site-packages/theano/gpuarray/__init__.py", line 164, in <module>
    use(config.device)
  File "/Users/np/miniconda3/lib/python3.6/site-packages/theano/gpuarray/__init__.py", line 151, in use
    init_dev(device)
  File "/Users/np/miniconda3/lib/python3.6/site-packages/theano/gpuarray/__init__.py", line 68, in init_dev
    context.cudnn_handle = dnn._make_handle(context)
  File "/Users/np/miniconda3/lib/python3.6/site-packages/theano/gpuarray/dnn.py", line 86, in _make_handle
    raise RuntimeError("error creating cudnn handle")
RuntimeError: error creating cudnn handle

With sudo I get

ERROR (theano.gpuarray): Could not initialize pygpu, support disabled
Traceback (most recent call last):
  File "/Users/np/miniconda3/lib/python3.6/site-packages/theano/gpuarray/__init__.py", line 164, in <module>
    use(config.device)
  File "/Users/np/miniconda3/lib/python3.6/site-packages/theano/gpuarray/__init__.py", line 151, in use
    init_dev(device)
  File "/Users/np/miniconda3/lib/python3.6/site-packages/theano/gpuarray/__init__.py", line 66, in init_dev
    avail = dnn.dnn_available(name)
  File "/Users/np/miniconda3/lib/python3.6/site-packages/theano/gpuarray/dnn.py", line 175, in dnn_available
    if not dnn_present():
  File "/Users/np/miniconda3/lib/python3.6/site-packages/theano/gpuarray/dnn.py", line 158, in dnn_present
    dnn_present.avail, dnn_present.msg = _dnn_check_version()
  File "/Users/np/miniconda3/lib/python3.6/site-packages/theano/gpuarray/dnn.py", line 131, in _dnn_check_version
    v = version()
  File "/Users/np/miniconda3/lib/python3.6/site-packages/theano/gpuarray/dnn.py", line 339, in version
    profile=False)
  File "/Users/np/miniconda3/lib/python3.6/site-packages/theano/compile/function.py", line 326, in function
    output_keys=output_keys)
  File "/Users/np/miniconda3/lib/python3.6/site-packages/theano/compile/pfunc.py", line 486, in pfunc
    output_keys=output_keys)
  File "/Users/np/miniconda3/lib/python3.6/site-packages/theano/compile/function_module.py", line 1795, in orig_function
    defaults)
  File "/Users/np/miniconda3/lib/python3.6/site-packages/theano/compile/function_module.py", line 1661, in create
    input_storage=input_storage_lists, storage_map=storage_map)
  File "/Users/np/miniconda3/lib/python3.6/site-packages/theano/gof/link.py", line 699, in make_thunk
    storage_map=storage_map)[:3]
  File "/Users/np/miniconda3/lib/python3.6/site-packages/theano/gof/vm.py", line 1047, in make_all
    impl=impl))
  File "/Users/np/miniconda3/lib/python3.6/site-packages/theano/gof/op.py", line 935, in make_thunk
    no_recycling)
  File "/Users/np/miniconda3/lib/python3.6/site-packages/theano/gof/op.py", line 839, in make_c_thunk
    output_storage=node_output_storage)
  File "/Users/np/miniconda3/lib/python3.6/site-packages/theano/gof/cc.py", line 1190, in make_thunk
    keep_lock=keep_lock)
  File "/Users/np/miniconda3/lib/python3.6/site-packages/theano/gof/cc.py", line 1131, in __compile__
    keep_lock=keep_lock)
  File "/Users/np/miniconda3/lib/python3.6/site-packages/theano/gof/cc.py", line 1586, in cthunk_factory
    key=key, lnk=self, keep_lock=keep_lock)
  File "/Users/np/miniconda3/lib/python3.6/site-packages/theano/gof/cmodule.py", line 1159, in module_from_key
    module = lnk.compile_cmodule(location)
  File "/Users/np/miniconda3/lib/python3.6/site-packages/theano/gof/cc.py", line 1489, in compile_cmodule
    preargs=preargs)
  File "/Users/np/miniconda3/lib/python3.6/site-packages/theano/gof/cmodule.py", line 2325, in compile_str
    return dlimport(lib_filename)
  File "/Users/np/miniconda3/lib/python3.6/site-packages/theano/gof/cmodule.py", line 302, in dlimport
    rval = __import__(module_name, {}, {}, [module_name])
ImportError: dlopen(/Users/np/.theano/compiledir_Darwin-16.5.0-x86_64-i386-64bit-i386-3.6.1-64/tmp3r02thlc/m3d1cf20adb1014f04986e6a344a55bde.so, 2): Library not loaded: @rpath/libcudnn.6.dylib
  Referenced from: /Users/np/.theano/compiledir_Darwin-16.5.0-x86_64-i386-64bit-i386-3.6.1-64/tmp3r02thlc/m3d1cf20adb1014f04986e6a344a55bde.so
  Reason: image not found

I know this is a problem with dynamic linking but I can't figure out why it isn't working.

I have tried the following:

• disabled SIP
• Reinstalled CUDA and cudnn probably around 10 times now
  • cudnn files are in both /Developer/NVIDIA/CUDA-8.0/lib and /usr/local/cuda/lib
• Reinstalled python, conda, theano, pygpu also around 10 times now
• Compiled libgpuarray from scratch
• Rebooted few times to make sure it's not that
• Executed update_dyld_shared_cache to see if it was a cache issue
• Tried to link libcudnn.6.dylib with install_name_tool to pygpu .so's but didn't do anything

Here are my paths from .zshrc:

export CUDA_HOME=/usr/local/cuda
export DYLD_LIBRARY_PATH=/usr/local/cuda/lib:/usr/local/cuda/extras/CUPTI/lib
export LD_LIBRARY_PATH=$DYLD_LIBRARY_PATH
export PATH=$CUDA_HOME/bi:$DYLD_LIBRARY_PATH:$PATH

.theanorc

[global]
floatX = float32
device = cuda
force_device = True
allow_gc = False
optimizer_including=cudnn

[cuda]
root = /usr/local/cuda

[dnn]
enabled = True
include_path=/usr/local/cuda/include
library_path=/usr/local/cuda/lib

Version numbers:

• macOS 10.12.4
• CUDA 8.0 and cudnn 6.0
• Python 3.6.1
• Theano 0.9.0
• libgpuarray 0.6.4
• pygpu 0.6.4

Answer 1

I was finally able to crack this by moving libcudnn* files to /usr/lib. I have absolutely no idea why pygpu wasn't able to work with /usr/local/cuda/lib. Without sudo I still get the RuntimeError: error creating cudnn handle error but now it works at least with sudo.

Though now I'm fighting with Segmentation fault: 11 but I'm guessing that's a different issue. Theano is an absolute...