Numbapro stopped working on the gpu

Question

I'm trying to run the following program:

import numpy as np
from timeit import default_timer as timer
from numbapro import vectorize


@vectorize(["float32(float32, float32)"], target='gpu')
def VectorAdd(a,b):
        return a + b

def main():
        N = 32e6

        A = np.ones(N, dtype=np.float32)
        B = np.ones(N, dtype=np.float32)
        C = np.zeros(N, dtype=np.float32)


        start = timer()
        C = VectorAdd(A, B, C)
        vectoradd_time = timer() - start

        print "C[:5] = " + str(C[:5])
        print "C[-5:] = " + str(C[-5:])

        print "VectorAdd took %f seconds" % vectoradd_time


if __name__ == '__main__':
        main()

The first time I ran it it worked fine. But then I tried to install nvprof and some additional libraries and since then I get the following error:

Traceback (most recent call last):
  File "test.py", line 6, in <module>
    @vectorize(["float32(float32, float32)"], target='gpu')
  File "/home/matthias/anaconda/lib/python2.7/site-packages/numba/npyufunc/decorators.py", line 68, in wrap
    vec.add(fty)
  File "/opt/anaconda1anaconda2anaconda3/lib/python2.7/site-packages/numbapro/cudavec/vectorizers.py", line 78, in add
  File "/home/matthias/anaconda/lib/python2.7/site-packages/numba/cuda/decorators.py", line 89, in kernel_jit
    kernel.bind()
  File "/home/matthias/anaconda/lib/python2.7/site-packages/numba/cuda/compiler.py", line 234, in bind
    self._func.get()
  File "/home/matthias/anaconda/lib/python2.7/site-packages/numba/cuda/compiler.py", line 181, in get
    cuctx = get_context()
  File "/home/matthias/anaconda/lib/python2.7/site-packages/numba/cuda/cudadrv/devices.py", line 126, in get_context
    return _get_device(devnum=devnum).context
  File "/home/matthias/anaconda/lib/python2.7/site-packages/numba/cuda/cudadrv/devices.py", line 118, in _get_device
    _gpustack.push(get_gpu(devnum))
  File "/home/matthias/anaconda/lib/python2.7/site-packages/numba/cuda/cudadrv/devices.py", line 108, in get_gpu
    return gpus[i]
  File "/home/matthias/anaconda/lib/python2.7/site-packages/numba/cuda/cudadrv/devices.py", line 34, in __getitem__
    return self._gpus[item]
  File "/home/matthias/anaconda/lib/python2.7/site-packages/numba/cuda/cudadrv/devices.py", line 23, in _gpus
    self._tls.gpus = self._init_gpus()
  File "/home/matthias/anaconda/lib/python2.7/site-packages/numba/cuda/cudadrv/devices.py", line 28, in _init_gpus
    for num in range(driver.get_device_count()):
  File "/home/matthias/anaconda/lib/python2.7/site-packages/numba/cuda/cudadrv/driver.py", line 256, in get_device_count
    self.cuDeviceGetCount(byref(count))
  File "/home/matthias/anaconda/lib/python2.7/site-packages/numba/cuda/cudadrv/driver.py", line 201, in __getattr__
    self.initialize()
  File "/home/matthias/anaconda/lib/python2.7/site-packages/numba/cuda/cudadrv/driver.py", line 182, in initialize
    raise CudaSupportError("Error at driver init: \n%s:" % e)
numba.cuda.cudadrv.error.CudaSupportError: Error at driver init: 
Call to cuInit results in CUDA_ERROR_NO_DEVICE:

Uninstalling the libraries hasn't helped.

Answer 1

I also experienced a similar issue in the past week. I upgraded my Anaconda environment from 2.7 to 3.4 and my code using numbapro started to fail with the same "Call to cuInit results in CUDA_ERROR_NO_DEVICE".

I'm not sure if my upgrade is the same cause as yours, but I mention just-in-case. (I should also mention I'm on Ubuntu using an ARM processor.)

I downloaded the cuda_6.5.14_linux_aarch64_native.run installer, rebooted into Ubuntu command line, and installed these latest drivers. After rebooting into X, I open iPython and run the following succesfully:

import numbapro
numbapro.check_cuda()

I hope this might help.

Answer 2

While it is not related to the error you are currently receiving, please note that there is also an error in your code on the line that calls VectorAdd.

You have:

C = VectorAdd(A, B, C)

It should be:

C = VectorAdd(A, B)