Sagemaker ml.g5.2xlarge instances not working as desired due to nvidia-drivers issue

Question

Over the weekend my sagemaker ml.g5.2xlarge started failing with the following errors: 
-> RuntimeError: No CUDA GPUs are available
-> NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

Answer

Hi,

I would suggest that you read 
* https://forums.developer.nvidia.com/t/nvidia-smi-has-failed-because-it-couldnt-communicate-with-the-nvidia-driver-make-sure-that-the-latest-nvidia-driver-is-installed-and-running/197141  
* https://forums.developer.nvidia.com/t/help-nvidia-smi-has-failed-because-it-couldnt-communicate-with-the-nvidia-driver-make-sure-that-the-latest-nvidia-driver-is-installed-and-running/260297/2

to try to find solutions to your issue.

Best,

Didier

Sagemaker ml.g5.2xlarge instances not working as desired due to nvidia-drivers issue

Relevant content