GPU Instance (g4dn.2xlarge) CUDA Driver Initialization Issue in Docker Container

Question

I am trying to run a container inside an AWS g4dn 2xlarge instance. For that I have configured the machine and the docker having:

nvidia-smi:
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.163.01             Driver Version: 550.163.01     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Tesla T4                       On  |   00000000:00:1E.0 Off |                    0 |
| N/A   25C    P8             15W /   70W |       1MiB /  15360MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

nvcc -V:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Mar_28_02:18:24_PDT_2024
Cuda compilation tools, release 12.4, V12.4.131
Build cuda_12.4.r12.4/compiler.34097967_0

Dockerfile:
FROM nvidia/cuda:12.4.1-base-ubuntu22.04

ENV DEBIAN_FRONTEND=noninteractive
ENV TZ=UTC

# Install Python 3.12 and required packages
RUN apt-get update && apt-get install -y --no-install-recommends \
    software-properties-common \
    && add-apt-repository ppa:deadsnakes/ppa \
    && apt-get update && apt-get install -y --no-install-recommends \
    python3.12 \
    python3.12-venv \
    python3.12-dev \
    python3-pip \
    git \
    libgl1-mesa-glx \
    libglib2.0-0 \
    libxext6 \
    libsm6 \
    libxrender1 \
    libgomp1 \
    # Add required NVIDIA dependencies
    # nvidia-cuda-toolkit \
    && rm -rf /var/lib/apt/lists/*

# Set Python 3.12 as default python
RUN update-alternatives --install /usr/bin/python python /usr/bin/python3.12 1
RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.12 1

# Set working directory
WORKDIR /app

# Install uv and project dependencies (excluding torch)
COPY requirements.txt ./

# Set NVIDIA environment variables
ENV NVIDIA_DRIVER_CAPABILITIES=compute,utility,video
ENV CUDA_VISIBLE_DEVICES=all
ENV NVIDIA_VISIBLE_DEVICES=all
ENV LD_LIBRARY_PATH=/usr/local/cuda/lib64:${LD_LIBRARY_PATH}

# Ensure Python outputs are not buffered
ENV PYTHONUNBUFFERED=1

COPY run.py .
COPY run2.py .

# Use exec form and specify the app entrypoint
CMD ["tail", "-f", "/dev/null"]

Docker run command:
docker run -it --rm --gpus all ai-pipeline bash

Inside docker:

# Ensure I can do anything with pip
python -m ensurepip --upgrade
pip install torch==2.5.1

# Running run.py and run2.py as they are code to check GPU access

root@c0c456bb9a32:/app# python run.py 
/usr/local/lib/python3.12/dist-packages/torch/_subclasses/functional_tensor.py:295: UserWarning: Failed to initialize NumPy: No module named 'numpy' (Triggered internally at ../torch/csrc/utils/tensor_numpy.cpp:84.)
  cpu = _conversion_method_template(device=torch.device("cpu"))
CUDA available: False
GPU count: 0
root@c0c456bb9a32:/app# python run2.py 
/usr/local/lib/python3.12/dist-packages/torch/_subclasses/functional_tensor.py:295: UserWarning: Failed to initialize NumPy: No module named 'numpy' (Triggered internally at ../torch/csrc/utils/tensor_numpy.cpp:84.)
  cpu = _conversion_method_template(device=torch.device("cpu"))
PyTorch version: 2.5.1+cu124
CUDA compiled version: 12.4
CUDA available: False
CUDA device count: 0

The strangest thing is that doing a quick env in the host machine with torch (2.8.0 with CUDA 12.8) gives:

(venv_prueba) ubuntu@ip-10-68-0-185:~$ python run2.py 
/home/ubuntu/venv_prueba/lib/python3.10/site-packages/torch/_subclasses/functional_tensor.py:279: UserWarning: Failed to initialize NumPy: No module named 'numpy' (Triggered internally at /pytorch/torch/csrc/utils/tensor_numpy.cpp:81.)
  cpu = _conversion_method_template(device=torch.device("cpu"))
PyTorch version: 2.8.0+cu128
CUDA compiled version: 12.8
CUDA available: True
CUDA device count: 1

What can this access error be about?

Thanks in advance.

MikeLim · Answer

You need to install and configure NVIDIA Container Toolkit in order for docker to use NVIDIA GPU. You can refer to this [article](https://repost.aws/articles/ARWGxLArMBQ4y1MKoSHTq3gQ/install-nvidia-gpu-driver-cuda-toolkit-nvidia-container-toolkit-on-amazon-ec2-instances-running-ubuntu-linux) for installation and configuration instructions.

I would suggest you consider using [AWS Deep Learning AMI (DLAMI)](https://aws.amazon.com/ai/machine-learning/amis/) which are available at no additional charge and comes with NVIDIA driver, Python, NVIDIA Container toolkit pre-installed. You can try [AWS Deep Learning OSS AMI GPU PyTorch 2.7 (Ubuntu 22.04)](https://docs.aws.amazon.com/dlami/latest/devguide/aws-deep-learning-ami-gpu-pytorch-2.7-ubuntu-22-04.html).

For guidance using DLAMI and installing graphical desktop, you can refer to [Deep Learning graphical desktop on Ubuntu Linux with AWS Deep Learning AMI (DLAMI)](https://repost.aws/articles/AR6RrDeUL1Tq6R8TgDs59iEA/deep-learning-graphical-desktop-on-ubuntu-linux-with-aws-deep-learning-ami-dlami).

re:Post Agent · Answer

Based on your description, you're experiencing a CUDA driver initialization issue in your Docker container despite having a properly configured host system. Here are several potential solutions to resolve this issue:

1. **NVIDIA Container Toolkit Configuration**:
Make sure the NVIDIA Container Toolkit is properly installed and configured on your host system. You should run:
```
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
```
This ensures Docker is properly configured to work with NVIDIA GPUs.

2. **Environment Variables**:
While you've included some environment variables in your Dockerfile, try adding these when running your container:
```
-e NVIDIA_VISIBLE_DEVICES=all -e NVIDIA_DRIVER_CAPABILITIES=all
```

3. **CUDA Version Compatibility**:
There might be a version mismatch between your host CUDA (12.4) and the container. Try using a PyTorch image that matches your host CUDA version more closely.

4. **Test with NVIDIA CUDA Container**:
As a diagnostic step, try running an official NVIDIA CUDA container to isolate whether the issue is specific to your custom container:
```
docker run --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi
```
If this works, the issue is specific to your container configuration.

5. **Install numpy**:
Your error logs show a warning about missing numpy. While this isn't directly related to the CUDA issue, install numpy in your container to eliminate this warning:
```
pip install numpy
```

6. **Docker Runtime Check**:
Verify that Docker is using the NVIDIA runtime by checking your Docker daemon configuration.

7. **Install CUDA Toolkit in Container**:
Your Dockerfile has the CUDA base image, but you might need the full CUDA toolkit inside the container. Consider uncommenting the `nvidia-cuda-toolkit` line in your Dockerfile.

The fact that PyTorch can access the GPU on the host but not in the container suggests that the container configuration is missing something needed for proper CUDA initialization.  
**Sources**  
[GPU Instance (g4dn.xlarge) CUDA Driver Initialization Issue in Docker Container | AWS re:Post](https://repost.aws/questions/QUW2UQ3CMuQiS0AgkHkc66Gg/gpu-instance-g4dn-xlarge-cuda-driver-initialization-issue-in-docker-container)  
[GPU on g4dn.xlarge is not active / responding anymore | AWS re:Post](https://repost.aws/questions/QUyK_YnuhRTqaKMXzj_hQlKg/gpu-on-g4dn-xlarge-is-not-active-responding-anymore)  
[How do I install NVIDIA GPU driver, CUDA Toolkit, NVIDIA Container Toolkit on Amazon EC2 instances running Amazon Linux 2 (AL2)? | AWS re:Post](https://repost.aws/articles/ARR29omO1-S5OfOPQKD904Jg/how-do-i-install-nvidia-gpu-driver-cuda-toolkit-nvidia-container-toolkit-on-amazon-ec2-instances-running-amazon-linux-2-al2)  
[Install NVIDIA GPU driver, CUDA Toolkit, NVIDIA Container Toolkit on Amazon EC2 instances running Amazon Linux 2 (AL2) | AWS re:Post](https://repost.aws/articles/ARR29omO1-S5OfOPQKD904Jg/install-nvidia-gpu-driver-cuda-toolkit-nvidia-container-toolkit-on-amazon-ec2-instances-running-amazon-linux-2-al2)

GPU Instance (g4dn.2xlarge) CUDA Driver Initialization Issue in Docker Container

Install Python 3.12 and required packages

Set Python 3.12 as default python

Set working directory

Install uv and project dependencies (excluding torch)

Set NVIDIA environment variables

Ensure Python outputs are not buffered

Use exec form and specify the app entrypoint

Ensure I can do anything with pip

Running run.py and run2.py as they are code to check GPU access

Add your answer

Relevant content