Theano for Keras: Failed to compile cuda_ndarray.cu: libcublas.so.7.5: cannot open shared object file: No such file or directory

Question

I'm trying to use Keras with the Theano backend inside a virtual environment. The error I'm getting is:

ERROR (theano.sandbox.cuda): Failed to compile cuda_ndarray.cu: libcublas.so.7.5: cannot open shared object file: No such file or directory

shown here:

(hyperproj) $ python
>> import theano
ERROR (theano.sandbox.cuda): Failed to compile cuda_ndarray.cu: libcublas.so.7.5: cannot open shared object file: No such file or directory

WARNING (theano.sandbox.cuda): The cuda backend is deprecated and will be removed in the next release (v0.10).  Please switch to the gpuarray backend. You can get more information about how to switch at this URL:
 https://github.com/Theano/Theano/wiki/Converting-to-the-new-gpu-back-end%28gpuarray%29

WARNING (theano.sandbox.cuda): CUDA is installed, but device gpu0 is not available  (error: cuda unavailable)

This is what I've done so far:

$ screen
$ conda create -n hyperproj python=2.7 anaconda
$ source activate hyperproj
(hyperproj) $ conda install numpy
(hyperproj) $ conda install keras
(hyperproj) $ conda install theano
(hyperproj) $ cd $HOME
(hyperproj) $ cd .keras/
(hyperproj) $ vim keras.json

This is what keras.json looks like now (I changed the tensorflow backend to the theano backend):

{
    "image_dim_ordering": "th",
    "epsilon": 1e-07,
    "floatx": "float32",
    "backend": "theano"
}

I need to modify (or create) my .theanorc file:

(hyperproj) $ vim ~/.theanorc

After several tutorials, it looks like this:

[global]
floatX=float32
device=gpu0

[nvcc]
fastmath=True

[cuda]
root = /usr/local/cuda-7.5

Someone said I may need to check my .bashrc file and modify my $PATH. Here's what that looks like now (after, for example, this):

# Added by Canopy installer on 2016-06-20
# VIRTUAL_ENV_DISABLE_PROMPT can be set to '' to make bashprompt show that Canopy is active, otherwise 1
VIRTUAL_ENV_DISABLE_PROMPT=1 source /mnt/data/user/pkgs/enthought/canopy-1.5.1/bin/activate

# added by Anaconda2 2.5.0 installer
export PATH="/export/mlrg/user/anaconda2/bin:$PATH"

# added by Anaconda2 4.1.1 installer
export PATH="/mnt/data/user/pkgs/anaconda2/bin:$PATH"

#added to fix the theano libcudnn issue
export PATH=/usr/local/cuda/bin:/usr/local/cuda/lib64:$PATH
export LD_LIBRARY_PATH="/usr/local/cuda/lib64/:$LD_LIBRARY_PATH

I have CUDA 7.5, Ubuntu 14.04, g++ version 4.8.4.

How can I resolve the error?

Answer 1

Strangely, the problem was solved by logging out and logging back in again. When I logged in, I did this:

$ screen
$ source activate hyperproj
(hyperproj) $ which nvcc #gave me /usr/local/cuda/bin/nvcc
(hyperproj) $ cd /usr/local/cuda/lib64 #checked that I have libcublas.so
(hyperproj) $ cd $HOME
(hyperproj) $ python
Python 2.7.13 |Anaconda custom (64-bit)| (default, Dec 20 2016, 23:09:15) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import keras
Using Theano backend.
WARNING (theano.sandbox.cuda): The cuda backend is deprecated and will be removed in the next release (v0.10).  Please switch to the gpuarray backend. You can get more information about how to switch at this URL:
 https://github.com/Theano/Theano/wiki/Converting-to-the-new-gpu-back-end%28gpuarray%29

Using gpu device 0: GeForce GTX TITAN Black (CNMeM is disabled, cuDNN not available)