Install NVIDIA GPU driver, CUDA Toolkit, NVIDIA Container Toolkit on Amazon EC2 instances running RHEL/Rocky Linux/AlmaLinux 8/9/10
Steps to install NVIDIA driver, CUDA Toolkit, NVIDIA Container Toolkit, and other NVIDIA software from NVIDIA repository on RHEL/Rocky Linux/AlmaLinux 8/9/10 (x86_64/arm64)
Overview
This article suggests how to install NVIDIA Data Center GPU Driver, CUDA Toolkit, NVIDIA Container Toolkit and other NVIDIA software directly from NVIDIA repository on NVIDIA GPU EC2 instances running RHEL (Red Hat Enterprise Linux) , Rocky Linux or AlmaLinux.
Note that by using this method, you agree to NVIDIA Driver License Agreement, End User License Agreement and other related license agreement. If you are doing development, you may want to register for NVIDIA Developer Program.
This article applies to RHEL/Rocky Linux/AlmaLinux on AWS only. Similar articles are available for AL2, AL2023, Ubuntu and Windows.
This article install NVIDIA Tesla driver which does not support G6f and Gr6f instance types
Other Options
If you need AMIs preconfigured with NVIDIA GPU driver, CUDA, other NVIDIA software, and optionally PyTorch or TensorFlow framework, consider AWS Deep Learning AMIs. Refer to Release notes for DLAMIs for currently supported options.
Refer to NVIDIA drivers for your Amazon EC2 instance for NVIDIA driver install options and NVIDIA Driver Installation Guide for Tesla driver installation instructions.
For container workloads, consider Amazon ECS-optimized Linux AMIs and Amazon EKS optimized AMIs
Note: instructions in this article are not applicable to pre-built AMIs.
Custom ECS GPU-optimized AMI
If you wish to build your own custom Amazon ECS GPU-optimized AMI, install NVIDIA driver, Docker, NVIDIA container toolkit and other related software, and refer to How do I create and use custom AMIs in Amazon ECS?
About CUDA toolkit
As CUDA driver is part of NVIDIA GPU driver, CUDA Toolkit is generally optional when GPU instance is used to run applications (as opposed to develop applications) as the CUDA application typically packages (by statically or dynamically linking against) the CUDA runtime and libraries needed.
System Requirements
NVIDIA CUDA supports the following platforms
- AlmaLinux 10 (x86_64)
- AlmaLinux 9 (x86_64)
- AlmaLinux 8 (x86_64)
- Red Hat Enterprise Linux (RHEL) 10 (x86_64 and arm64)
- Red Hat Enterprise Linux (RHEL) 9 (x86_64 and arm64)
- Red Hat Enterprise Linux (RHEL) 8 (x86_64 and arm64)
- Rocky Linux 10 (x86_64)
- Rocky Linux 9 (x86_64)
- Rocky Linux 8 (x86_64)
While it may work, NVIDIA do not support Rocky Linux and AlmaLinux on arm64 architecture or other RHEL compatible Linux OSs. as Refer to Driver installation guide for supported kernel versions, compilers and libraries.
Prerequisites
Go to Service Quotas console of your desired Region to verify On-Demand Instance quota value of your desired instance type:
- G instance types: Running On-Demand G and VT instances
- P instance types: Running On-Demand P instances
Request quota increase if the assigned value is less than vCPU count of your desired EC2 instance size. Do not proceed until your applied quota value is equal or higher than your instance type vCPU count
Prepare AlmaLinux / Rocky Linux / RHEL
Launch a new NVIDIA GPU instance preferably with at least 20 GB storage and connect to the instance
Update OS, add EPEL repository, install DKMS, kernel headers and development packages
sudo dnf update -y OS_VERSION=$(. /etc/os-release;echo $VERSION_ID | sed -e 's/\..*//g') if ( cat /etc/os-release | grep -q Red ); then sudo subscription-manager repos --enable=rhel-$OS_VERSION--for-$(arch)-appstream-rpms sudo subscription-manager repos --enable=rhel-$OS_VERSION--for-$(arch)-baseos-rpms sudo subscription-manager repos --enable=codeready-builder-for-rhel-$OS_VERSION-$(arch)-rpms elif ( echo $OS_VERSION | grep -q 8 ); then sudo dnf config-manager --set-enabled powertools else sudo dnf config-manager --set-enabled crb fi sudo dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-$OS_VERSION.noarch.rpm sudo dnf clean expire-cache sudo dnf install -y dkms kernel-devel kernel-modules-extra amazon-ec2-utils unzip gcc make vulkan-devel libglvnd-devel elfutils-libelf-devel sudo systemctl daemon-reload sudo systemctl enable dkms
Restart your EC2 instance if kernel is updated
sudo reboot
Add NVIDIA repository
Configure Network Repo installation
DISTRO=$(. /etc/os-release;echo rhel$VERSION_ID | sed -e 's/\..*//g') if (arch | grep -q x86); then ARCH=x86_64 else ARCH=sbsa fi sudo dnf config-manager --add-repo http://developer.download.nvidia.com/compute/cuda/repos/$DISTRO/$ARCH/cuda-$DISTRO.repo sudo dnf clean expire-cache
If you are installing from AWS China Region, you may be able to replace repository source from https://developer.download.nvidia.com to https://developer.download.nvidia.cn
if (ec2-metadata -z | grep cn-); then sudo sed -i "s/nvidia\.com/nvidia\.cn/g" /etc/yum.repos.d/cuda-rhel*.repo sudo dnf clean expire-cache fi
Install NVIDIA Driver
To install latest Tesla driver
if ( sudo dnf module list | grep nvidia-driver | grep -q open-dkms); then sudo dnf module enable -y nvidia-driver:open-dkms fi sudo dnf install -y nvidia-open if ( sudo dnf search nvidia-xconfig | grep -q nvidia-xconfig); then sudo dnf install -y nvidia-xconfig fi
AlmaLinux/RHEL/Rocky 8/9 only: To install a specific driver branch, e.g. R580 LTSB
sudo dnf module enable -y nvidia-driver:580-open sudo dnf install -y nvidia-open if ( sudo dnf search nvidia-xconfig | grep -q nvidia-xconfig); then sudo dnf install -y nvidia-xconfig fi
Refer to Version Locking if you want to lock NVIDIA driver branch.
The above install open-source GPU kernel module which is recommended by NVIDIA (and is different from Nouveau open-source driver). Refer to Driver Installation Guide about NVIDIA Kernel Modules and installation options.
Alternatively, pre-compiled RHEL kernel modules may be available
sudo dnf module enable -y nvidia-driver:latest sudo dnf install -y nvidia-driver nvidia-driver-cuda sudo dnf install -y nvidia-xconfig
CUDA Compatability
If you did not install the latest NVIDIA driver and need to support applications built on newer CUDA Toolkits, you can install a special package called "CUDA Compatability Package". Refer to NVIDIA CUDA Compatability documentation for more details
P instance
If you are using a P instance with multiple GPUs, you may need to install Fabric Manager. Refer to UFM (Unified Fabric Manager) section below for details.
Optional: Compute-only and Desktop Installation
NVIDIA supports custom installation method which supports the following configurations:
- Desktop: Contains all the X/Wayland drivers and libraries to allow running a GPU with power management enabled on a desktop system but does not include any CUDA component
- Compute-only / headless: Contains everything required to run CUDA applications on a GPU system where the GPU is not used to drive a display
- Desktop and Compute: canonical way of installing the driver, with every possible library and display component. This might be required in cross functional combinations, for CUDA-accelerated video encoding/decoding.
To install for the above cases:
- Desktop only:
sudo dnf install -y nvidia-driver kmod-nvidia-open-dkms - Compute-only/headless:
sudo dnf install -y nvidia-driver-cuda kmod-nvidia-open-dkms - Desktop and Compute:
sudo dnf install -y nvidia-open
Refer to NVIDIA Driver Installation Guide for more information.
Verify
Restart your instance
nvidia-smi
Output should be similar to below
Sun Dec 21 05:32:21 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 590.44.01 Driver Version: 590.44.01 CUDA Version: 13.1 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 |
| 0% 31C P8 16W / 300W | 0MiB / 23028MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
Refer to section Verify installation integrity on steps to verify CUDA driver integrity.
Optional: CUDA Toolkit
To install latest CUDA Toolkit
sudo dnf install -y cuda-toolkit
To install a specific series, e.g. 12.x
sudo dnf install -y cuda-toolkit-12
To install a specific version, e.g. 12.9
sudo dnf install -y cuda-toolkit-12-9
Refer to CUDA Toolkit documentation about supported platforms and installation options
Verify
/usr/local/cuda/bin/nvcc -V
Output should be similar to below
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Fri_Nov__7_07:23:37_PM_PST_2025
Cuda compilation tools, release 13.1, V13.1.80
Build cuda_13.1.r13.1/compiler.36836380_0
Post-installation Actions
Refer to NVIDIA CUDA Installation Guide for Linux for post-installation actions before CUDA Toolkit can be used. For example, you may want to modify your PATH and LD_LIBRARY_PATH environment variables to include /usr/local/cuda/bin and /usr/local/cuda/lib64 respectively
sed -i '$aexport PATH=$PATH:/usr/local/cuda/bin' ~/.bashrc sed -i '$aexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64' ~/.bashrc . ~/.bashrc
Optional: NVIDIA Container Toolkit
NVIDIA Container toolkit supports RHEL 8, 9 and 10 on both x86_64 and arm64. For arm64, use g5g.2xlarge or larger instance size as g5g.xlarge may cause failures due to the limited system memory.
To install latest NVIDIA Container Toolkit
if (! sudo dnf search nvidia | grep -q nvidia-container-toolkit); then sudo dnf config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo fi sudo dnf install -y nvidia-container-toolkit
Refer to NVIDIA Container toolkit documentation about supported platforms, prerequisites and installation options
Verify
nvidia-container-cli -V
Output should be similar to below
cli-version: 1.18.1
lib-version: 1.18.1
build date: 2025-11-24T14:45+0000
build revision: 889a3bb5408c195ed7897ba2cb8341c7d249672f
build compiler: gcc 4.8.5 20150623 (Red Hat 4.8.5-44)
build platform: x86_64
build flags: -D_GNU_SOURCE -D_FORTIFY_SOURCE=2 -DNDEBUG -std=gnu11 -O2 -g -fdata-sections -ffunction-sections -fplan9-extensions -fstack-protector -fno-strict-aliasing -fvisibility=hidden-Wall -Wextra -Wcast-align -Wpointer-arith -Wmissing-prototypes -Wnonnull -Wwrite-strings -Wlogical-op -Wformat=2 -Wmissing-format-attribute -Winit-self -Wshadow -Wstrict-prototypes -Wunreachable-code -Wconversion -Wsign-conversion -Wno-unknown-warning-option -Wno-format-extra-args -Wno-gnu-alignof-expression -Wl,-zrelro -Wl,-znow -Wl,-zdefs -Wl,--gc-sections
Container engine configuration
Refer to NVIDIA Container Toolkit documentation about container engine configuration.
Install and configure Docker
To install and configure docker.
if (cat /etc/os-release | grep -q Rocky); then USER="rocky" else USER="ec2-user" fi sudo dnf config-manager --add-repo https://download.docker.com/linux/rhel/docker-ce.repo sudo dnf clean expire-cache sudo dnf install -y docker-ce docker-ce-cli containerd.io sudo usermod -aG docker $USER sudo systemctl enable docker sudo nvidia-ctk runtime configure --runtime=docker sudo systemctl restart docker
Verify Docker engine configuration
To verify docker configuration
sudo docker run --rm --runtime=nvidia --gpus all public.ecr.aws/docker/library/rockylinux:9 nvidia-smi
Output should be similar to below
Unable to find image 'public.ecr.aws/docker/library/rockylinux:9' locally
9: Pulling from docker/library/rockylinux
446f83f14b23: Pull complete
Digest: sha256:d7be1c094cc5845ee815d4632fe377514ee6ebcf8efaed6892889657e5ddaaa6
Status: Downloaded newer image for public.ecr.aws/docker/library/rockylinux:9
Sun Dec 21 05:34:31 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 590.44.01 Driver Version: 590.44.01 CUDA Version: 13.1 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 |
| 0% 31C P8 16W / 300W | 0MiB / 23028MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
EC2 Install Script
You can use the below as install script (or user data) to install GPU driver and NVIDIA Container Toolkit on a new RHEL/Rocky Linux NVIDIA GPU instance preferably with latest patches applied and at least 20 GB storage,.
Remove # character (except the first line) if you wish to install CUDA toolkit
#!/bin/bash sudo dnf clean all OS_VERSION=$(. /etc/os-release;echo $VERSION_ID | sed -e 's/\..*//g') if ( cat /etc/os-release | grep -q Red ); then sudo subscription-manager repos --enable=rhel-$OS_VERSION--for-$(arch)-appstream-rpms sudo subscription-manager repos --enable=rhel-$OS_VERSION--for-$(arch)-baseos-rpms sudo subscription-manager repos --enable=codeready-builder-for-rhel-$OS_VERSION-$(arch)-rpms elif ( echo $OS_VERSION | grep -q 8 ); then sudo dnf config-manager --set-enabled powertools else sudo dnf config-manager --set-enabled crb fi sudo dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-$OS_VERSION.noarch.rpm sudo dnf clean expire-cache sudo dnf install -y kernel-devel kernel-modules-extra unzip gcc make vulkan-devel libglvnd-devel elfutils-libelf-devel sudo dnf install -y dkms amazon-ec2-utils sudo dnf install -y kernel-devel-$(uname -r) kernel-modules-extra-$(uname -r) sudo systemctl daemon-reload sudo systemctl enable dkms DISTRO=$(. /etc/os-release;echo rhel$VERSION_ID | sed -e 's/\..*//g') if (arch | grep -q x86); then ARCH=x86_64 else ARCH=sbsa fi sudo dnf config-manager --add-repo http://developer.download.nvidia.com/compute/cuda/repos/$DISTRO/$ARCH/cuda-$DISTRO.repo sudo dnf clean expire-cache if ( sudo dnf module list | grep nvidia-driver | grep -q open-dkms ); then sudo dnf module enable -y nvidia-driver:open-dkms fi sudo dnf install -y nvidia-open if ( sudo dnf search nvidia-xconfig | grep -q nvidia-xconfig ); then sudo dnf install -y nvidia-xconfig fi if (cat /etc/os-release | grep -q Rocky); then USER="rocky" else USER="ec2-user" fi # sudo dnf install -y cuda-toolkit # sed -i '$aexport PATH=$PATH:/usr/local/cuda/bin' /home/$USER/.bashrc # sed -i '$aexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64' /home/$USER/.bashrc sudo dnf config-manager --add-repo https://download.docker.com/linux/rhel/docker-ce.repo sudo dnf clean expire-cache sudo dnf install -y docker-ce docker-ce-cli containerd.io sudo systemctl enable docker sudo usermod -aG docker $USER if (! sudo dnf search nvidia | grep -q nvidia-container-toolkit ); then sudo dnf config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo sudo dnf clean expire-cache fi sudo dnf install -y nvidia-container-toolkit sudo nvidia-ctk runtime configure --runtime=docker sudo systemctl restart docker if ( ec2-metadata -t | grep -q " p[0-9]" ); then if ( sudo dnf module list | grep nvidia-driver | grep open-dkms | grep -q fm ); then sudo dnf module install -y nvidia-driver:open-dkms/fm else sudo dnf install -y nvidia-fabricmanager libnvidia-nscq libnvsdm nvidia-imex fi if ( ec2-metadata -t | grep -q " p[6-9]" ); then sudo dnf install -y nvlsm nvlink5 echo "ib_umad" | sudo tee -a /etc/modules-load.d/modules.conf sudo modprobe ib_umad fi sudo systemctl enable --now nvidia-fabricmanager fi sudo systemctl daemon-reload sudo systemctl enable dkms sudo reboot
On RHEL/Rocky 10, you may need to install Fabric Manager manually on P instance
Verify
Connect to your EC2 instance
nvidia-smi nvidia-container-cli -V sudo docker run --rm --runtime=nvidia --gpus all public.ecr.aws/docker/library/rockylinux:9 nvidia-smi
If used as user data, view /var/log/cloud-init-output.log to troubleshoot any installation issues.
Perform post-installation actions in order to use CUDA toolkit (if installed).
Verify installation integrity
NVIDIA driver and NVIDIA Container Toolkit
To verify integrity of installation, you can use CUDA samples container image to validate CUDA driver.
sudo docker run --rm --runtime=nvidia --gpus all nvcr.io/nvidia/k8s/cuda-sample:devicequery
Ensure you get Result = PASS output.
NVIDIA driver and CUDA Toolkit
If CUDA toolkit is installed, you can download, compile and run CUDA samples such as deviceQuery.
If you are using a P instance with multiple GPUs, you may need to install Fabric Manager. Refer to UFM (Unified Fabric Manager) section below for instructions
GUI (graphical desktop) remote access
If you need remote graphical desktop access, refer to Install GUI (graphical desktop) on Amazon EC2 instances running RHEL/Rocky Linux 8/9?
Note that this article installs NVIDIA Tesla driver (also know as NVIDIA Datacenter Driver), which is intended primarily for GPU compute workloads. If configured in xorg.conf, Tesla drivers support one display of up to 2560x1600 resolution.
GRID drivers provide access to four 4K displays per GPU and are certified to provide optimal performance for professional visualization applications. AMIs preconfigured with GRID drivers are available from AWS Marketplace. You can also consider using amazon-ec2-nice-dcv-samples CloudFormation templates to provision your own EC2 instances with either NVIDIA Tesla or GRID driver, Docker with NVIDIA Container Toolkit, graphical desktop environment and Amazon DCV remote display protocol server.
Other software
AWS CLI
To install AWS CLI (AWS Command Line Interface) v2 through Snap
sudo dnf install -y snapd sudo systemctl enable --now snapd snapd.socket sudo ln -s /var/lib/snapd/snap /snap sudo snap install aws-cli --classic
Verify
Log off and log in so that your PATH variables are updated correctly.
aws --version
Output should be similar to below
aws-cli/2.27.53 Python/3.13.4 Linux/5.14.0-570.39.1.el9_6.x86_64 exe/x86_64.rhel.9
SSM Agent
To install SSM agent for Session Manager access
if (arch | grep -q x86); then sudo dnf install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm else sudo dnf install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_arm64/amazon-ssm-agent.rpm fi
This requires EC2 instance to have attached IAM role with the AmazonSSMManagedInstanceCore managed policy
EC2 Instance Connect
To install EC2 Instance Connect for secure SSH access
cd /tmp if (arch | grep -q x86); then ARCH=amd64 else ARCH=arm64 fi if ( cat /etc/os-release | grep -q 8\. ); then curl -L -O https://amazon-ec2-instance-connect-us-west-2.s3.us-west-2.amazonaws.com/latest/linux_$ARCH/ec2-instance-connect.rhel8.rpm curl -L -O https://amazon-ec2-instance-connect-us-west-2.s3.us-west-2.amazonaws.com/latest/linux_amd64/ec2-instance-connect-selinux.noarch.rpm sudo dnf install -y ./ec2-instance-connect.rhel8.rpm ./ec2-instance-connect-selinux.noarch.rpm else curl -L -O https://amazon-ec2-instance-connect-us-west-2.s3.us-west-2.amazonaws.com/latest/linux_$ARCH/ec2-instance-connect.rpm curl -L -O https://amazon-ec2-instance-connect-us-west-2.s3.us-west-2.amazonaws.com/latest/linux_amd64/ec2-instance-connect-selinux.noarch.rpm sudo dnf install -y ./ec2-instance-connect.rpm ./ec2-instance-connect-selinux.noarch.rpm fi sudo systemctl restart sshd
Allow inbound SSH traffic in your security group
cuDNN (CUDA Deep Neural Network library)
To install cuDNN for the latest available CUDA version.
sudo dnf install -y zlib cudnn
Refer to cuDNN documentation about installation options and support matrix
NCCL (NVIDIA Collective Communication Library)
To install latest NCCL
sudo dnf install -y libnccl libnccl-devel libnccl-static
Refer to NCCL documentation about installation options
DCGM (Data Center GPU Manager)
To install DCGM
CUDA_VERSION=$(nvidia-smi | sed -E -n 's/.*CUDA Version: ([0-9]+)[.].*/\1/p') sudo dnf install --assumeyes \ --setopt=install_weak_deps=True \ datacenter-gpu-manager-4-cuda${CUDA_VERSION}
Refer to DCGM documentation for more information
Verify
dcgmi --version
Output should be similar to below
dcgmi version: 4.4.2
GDS (GPUDirect Storage)
To install NVIDIA Magnum IO GPUDirect® Storage (GDS)
sudo dnf install -y nvidia-gds
To install for a specific CUDA version, e.g. 13.0
sudo dnf install -y nvidia-gds-13-0
Reboot
Restart to load kernel module
sudo reboot
Verify
To verify module
lsmod | grep nvidia_fs
Output should be similar to below
nvidia_fs 323584 0
nvidia 11579392 3 nvidia_uvm,nvidia_fs,nvidia_modeset
To verify successful installation
sudo /usr/local/cuda/gds/tools/gdscheck -p
Output should be similar to below
GDS release version: 1.16.0.49
nvidia_fs version: 2.27 nvidia_fs minimum version: 2.12
Platform: x86_64
...
...
=========
GPU INFO:
=========
GPU index 0 NVIDIA A10G bar:1 bar size (MiB):32768 supports GDS, IOMMU State: Disabled
==============
PLATFORM INFO:
==============
IOMMU: disabled
Nvidia Driver Info Status: Supported(Nvidia Open Driver Installed)
Cuda Driver Version Installed: 13010
Platform: g5.xlarge, Arch: x86_64(Linux 6.12.0-124.21.1.el10_1.x86_64)
Platform verification succeeded
Refer to GDS documentation and Driver installation guide for more information
GDRCopy
Magnum IO GDRCopy packages for different CUDA versions can be installed from NVIDIA Developer download site. Alternatively, download and compile from Github
Restart your EC2 instance
sudo reboot
Verify
lsmod | grep gdr
Output should be similar to below
gdrdrv 28672 0
nvidia 14376960 7 nvidia_uvm,gdrdrv,nvidia_modeset
CUDA-X Libraries
NVIDIA repository also provides access to CUDA Math, Quantum and other libraries such as cuTENSOR, cuFFT and cuQuantum. Refer to NVIDIA site for more information
UFM (Unified Fabric Manager)
Eligibility
To determine if you need NVIDIA Unified Fabric Manager (UFM)
nvidia-smi -q -i 0 | grep Fabric -A2 | grep State
If State is N/A, you do not need Fabric Manager
State : N/A
If State is not N/A, install Fabric Manager as per next section
State : In Progress
Install
To install latest NVIDIA Unified Fabric Manager (UFM), NSCQ, NVSDM, IMEX for EC2 instances with NVIDIA NVLink.
if ( sudo dnf module list | grep nvidia-driver | grep open-dkms | grep -q fm ); then sudo dnf module install -y nvidia-driver:open-dkms/fm else sudo dnf install -y nvidia-fabricmanager libnvidia-nscq libnvsdm nvidia-imex fi sudo systemctl enable --now nvidia-fabricmanager
P6 instance
P6 instance requires NVLink Subnet Manager (NVLSM)
sudo dnf install -y nvlsm nvlink5 echo "ib_umad" | sudo tee -a /etc/modules-load.d/modules.conf sudo modprobe ib_umad sudo systemctl restart nvidia-fabricmanager
Refer to EC2 and NVIDIA documentation for up to date instructions.
Verify
nv-fabricmanager -v systemctl status nvidia-fabricmanager
Output should be similar to below
Fabric Manager version is : 590.44.01
● nvidia-fabricmanager.service - NVIDIA fabric manager service
Loaded: loaded (/usr/lib/systemd/system/nvidia-fabricmanager.service; enabled; preset: enabled)
Active: active (running) since ......... UTC; 1min 4s ago
Process: 22851 ExecStart=/usr/bin/nvidia-fabricmanager-start.sh --mode start (code=exited, status=0/SUCCESS)
Main PID: 22881 (nv-fabricmanage)
Tasks: 18 (limit: 3355442)
Memory: 38.1M
CPU: 633ms
CGroup: /system.slice/nvidia-fabricmanager.service
└─22881 /usr/bin/nv-fabricmanager -c /usr/share/nvidia/nvswitch/fabricmanager.cfg
.........compute.internal nv-fabricmanager[22881]: Starting nvidia-fabricmanager.service - NVIDIA fabric manager service...
.........compute.internal nv-fabricmanager[22881]: Detected Pre-NVL5 system
.........compute.internal nv-fabricmanager[22881]: Connected to 1 node.
.........compute.internal nv-fabricmanager[22881]: Successfully configured all the available NVSwitches to route GPU NVLink traffic. NVLink Peer-to-Peer support will be enabled once the GPUs are successfully registered with the NVLink fabric.
.........compute.internal nv-fabricmanager[22881]: Started "Nvidia Fabric Manager"
.........compute.internal nv-fabricmanager[22881]: Started nvidia-fabricmanager.service - NVIDIA fabric manager service.
To view GPU fabric registration status
nvidia-smi -q -i 0 | grep -i -A 2 Fabric
Output should be similar to below after GPUs have successfully registered
Fabric
State : Completed
Status : Success
Refer to Fabric Manager documentation for more information.
- Language
- English
Relevant content
asked 3 years ago
