I got an error Failed to initialize NVML: Driver/library version mismatch on a cloud virtual machine for unknown reasons, the system was working normally then suddenly crashed and reported such an error,
I'm very confused and don't know what is the cause, can someone with experience in this matter please help me, I want to know why I get such an error and is there any way to prevent it, thanks
As per this doc curated by Bright computing knowledge base the “Failed to initialize NVML: Driver/library version mismatch?” error generally means the CUDA Driver is still running an older release that is incompatible with the CUDA toolkit version currently in use.
Refer to this official document to get rid of all previous CUDA and NVIDIA driver files, follow the steps in the cuda linux installation guide and then reinstall.