gpu is not detected by pytorch for spacy NER

Question

I am trying to train a spacy3 NER model. I have GPU available on my machine with following specs:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.86.01    Driver Version: 515.86.01    CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |
| N/A   68C    P0    29W /  70W |    567MiB / 15360MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                           
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1352      C   python                            563MiB |
+-----------------------------------------------------------------------------+

while GPU is active/available and running through following commands:

>>>import torch
>>>torch.backends.cudnn.is_available()    
True

>>>torch.backends.cudnn.version()
8500

>>>torch.cuda.is_available()
True

>>>torch.zeros(1).cuda()
tensor([0.], device='cuda:0')

>>>torch.cuda.get_device_name()
'Tesla T4'

when I run the following command to train my spacy ner model,

 python3 -m spacy train configs/config.cfg --output training/ --path.train corpus/train.spacy --path.dev corpus/train.spacy --gpu-id 0

I am getting the following error:

 ValueError: No GPU devices detected

I have resolved version error and on top of it, GPU is available for normal tasks. I have gone through following stackoverflow questions but none of these is helpful.

GPU is not available for Pytorch

PyTorch is not finding GPU in Ubuntu

Why torch.cuda.is_available() returns False even after installing pytorch with cuda?

I am struck with this problem and unable to find a solution. Any sort of help is highly appreciated.

I am using following version of python and pytorch respectively

Python 3.10.6
1.13.1+cu117

Answer 1

Please check Whether you have set gpu_allocator = "pytorch" and seed = 0 in the system field of the config file.