AWS Builder Center is the official home for builders on AWS. Share and read what others are working on, follow people who inspire you, explore training and workshops, and find tools to support what you're building.
Beyond CloudWatch: GPU Observability for Training and Inference Workloads with NVIDIA DCGM Exporter
This article explains why Amazon CloudWatch cannot natively monitor NVIDIA GPU utilization, memory, thermal, or error metrics on EC2-based training and inference workloads — and provides a practical implementation guide for deploying NVIDIA DCGM Exporter with Amazon Managed Prometheus and Managed Grafana to close that gap.
The problem: CloudWatch doesn't see inside GPUs
If you run GPU-accelerated training or inference on AWS — whether on Amazon EC2 P-series instances, AWS ParallelCluster with Slurm, Amazon EKS with GPU node groups, or SageMaker HyperPod — you've likely noticed that Amazon CloudWatch gives you no GPU-specific metrics by default. CloudWatch tells you the instance is running and how busy its CPU is. It does not tell you:
- Whether the GPU's streaming multiprocessors are actually doing work
- How much GPU memory is used vs. free
- Whether thermal throttling is limiting performance
- Which training job is consuming which GPU
- Whether an ECC error is silently corrupting your gradients
For a $20–$100+/hr GPU fleet running training or inference at scale, this is a significant operational visibility gap. You're paying for GPU time but cannot verify it's being used effectively — or detect when it isn't.
This article explains what CloudWatch can and cannot see on GPU instances, introduces NVIDIA DCGM (Data Center GPU Manager) Exporter as the solution, describes the scenarios where it matters most, and provides a condensed implementation guide for deploying it on AWS.
What CloudWatch sees on a GPU instance
CloudWatch's standard EC2 metrics plus the optional CloudWatch Agent give you:
| Metric | Source | What it tells you |
|---|---|---|
CPUUtilization | EC2 hypervisor | CPU busy % (averaged across all vCPUs) |
NetworkIn/Out | EC2 hypervisor | Network bytes transferred |
EBSReadOps/WriteOps | EC2 hypervisor | Storage I/O |
mem_used_percent | CloudWatch Agent | RAM utilization (requires agent install) |
disk_used_percent | CloudWatch Agent | Disk utilization (requires agent install) |
StatusCheckFailed | EC2 hypervisor | Instance-level health (host or system failure) |
What's notably absent: anything about the GPU — utilization, memory, temperature, power, errors, NVLink bandwidth, tensor core activity. The GPU is opaque to CloudWatch.
Why doesn't CloudWatch collect GPU metrics natively?
NVIDIA GPU metrics are exposed through NVIDIA's proprietary NVML (NVIDIA Management Library) and DCGM APIs — not through the standard Linux /proc or /sys interfaces that the CloudWatch Agent reads. Collecting GPU telemetry requires a GPU-aware agent that speaks NVML/DCGM. AWS provides this in managed contexts (SageMaker HyperPod has built-in DCGM integration), but on self-managed EC2, ParallelCluster, and EKS, you deploy it yourself.
When the gap matters — practical scenarios
These are real operational situations where the CloudWatch-only visibility gap causes problems:
Training workloads (HPC / Slurm / ParallelCluster)
| # | Scenario | What CloudWatch shows | What you actually need |
|---|---|---|---|
| 1 | Distributed training run is 3× slower than expected | CPU% is low | GPU SM utilization, EFA RDMA bandwidth, per-job FSx throughput — identify whether the bottleneck is compute, network, or storage |
| 2 | Multi-node job failed at hour 4 of a 6-hour run — which node first, why? | Slurm exit code | Correlated XID/ECC errors, thermal throttle events, EFA link flaps — timestamped and cross-node |
| 3 | User requested 4 GPUs for 12 hours but spent the first 6 downloading data at 0% GPU utilization | You billed the same either way | GPU active-time vs. allocated-time reveals "allocated-but-idle" — silent waste |
| 4 | Which team or user consumed $100K of GPU time last month? | Total bill, no attribution | Per-job GPU-hours via Slurm accounting + DCGM labels |
| 5 | GPU training partition has 47 pending jobs — capacity exhaustion, fair-share issue, or one user monopolizing resources? | Nothing — you SSH and run squeue | Real-time queue depth per partition, per user, time-in-queue |
Inference workloads (EKS / ECS / self-managed)
| # | Scenario | What CloudWatch shows | What you actually need |
|---|---|---|---|
| 6 | Model serving latency spiked — is it GPU saturation, memory pressure, or a framework bug? | Pod CPU and memory (via Container Insights) | GPU SM occupancy, tensor core utilization, FB memory used — distinguish "GPU is maxed" from "GPU is waiting on something else" |
| 7 | One node in the inference fleet has degraded performance on random days | Nothing on the GPU | Per-device RDMA read/write, dropped packets, ECC errors — identifies a bad NIC or degrading GPU |
| 8 | Auto-scaler isn't scaling fast enough — but is the existing fleet actually saturated? | CPU-based scaling triggers don't apply | Custom metrics: DCGM_FI_DEV_GPU_UTIL as a scaling signal — scale on actual GPU utilization, not proxied via CPU |
The cross-cutting scenario (applies to both)
| # | Scenario | What CloudWatch shows | What you actually need |
|---|---|---|---|
| 9 | "The job stalled — was it the GPU, the auth system, the storage, or the scheduler?" | Three separate consoles, manual correlation | A single timeline where GPU activity, auth latency, storage throughput, and scheduler state are queryable together — the OpenTelemetry Collector layer that unifies DCGM + service metrics into one Prometheus/Grafana view |
The solution: NVIDIA DCGM Exporter + Amazon Managed Prometheus + Amazon Managed Grafana
NVIDIA DCGM (Data Center GPU Manager) is the official, supported interface for collecting GPU telemetry from NVIDIA data-center GPUs (A100, H100, H200, B200, and all P-series instances on AWS). The DCGM Exporter is an open-source component that exposes these metrics in Prometheus format at an HTTP endpoint — making them scrapable by any Prometheus-compatible system.
The AWS-recommended architecture for GPU observability on EC2 and ParallelCluster:
GPU Compute Node(s) Head Node / Monitoring
┌──────────────────────┐ ┌─────────────────────────────────┐
│ DCGM Exporter :9400 │──scrape─│ Prometheus (agent mode) │
│ Node Exporter :9100 │──scrape─│ remote_write (SigV4) │──── Amazon Managed
│ EFA Exporter :9101 │──scrape─│ │ Service for Prometheus
│ Slurm Exporter:8080 │──scrape─│ │ │
└──────────────────────┘ └─────────────────────────────────┘ │
▼
Amazon Managed Grafana
┌──────────────────────┐
│ DCGM Dashboard 12239 │
│ Jobs Dashboard │
│ EFA Dashboard │
│ Cluster Overview │
└──────────────────────┘
For EKS-based inference, the same pattern deploys as a Kubernetes DaemonSet:
kubectl / Helm
└── dcgm-exporter DaemonSet (nvidia-dcgm/dcgm-exporter chart)
└── scraped by ADOT Collector or Prometheus Operator
└── remote_write to AMP
└── visualized in AMG
Why this pattern (and not self-hosted Prometheus/Grafana)?
| Consideration | Self-hosted on head node | Managed (AMP + AMG) |
|---|---|---|
| Scaling across clusters | One Prometheus per cluster; no unified view | One AMP workspace spans all clusters |
| Operational burden | You patch, scale, backup the TSDB | AWS manages — zero ops |
| Retention | Disk-limited (weeks) | 150 days built-in, configurable |
| HA/Durability | Single-node SPOF | Multi-AZ, replicated |
| Cost | EC2 time (hidden in head-node bill) | ~$0.90/10M samples + $0.03/GB stored; $9/user/mo for Grafana |
| Best for | Single small cluster, budget-constrained | Multi-cluster fleet, production GPU workloads at scale |
For fleets larger than 1–2 clusters, the managed pattern is strongly preferred. For a single ParallelCluster in a dev/test context, self-hosted on the head node (per the AWS Prescriptive Guidance pattern) is a reasonable starting point.
Key DCGM metrics — what to focus on
DCGM exports ~35 metrics per GPU by default. Not all are equally useful. Here's what matters most for training and inference:
Utilization & efficiency ("Is the GPU actually working?")
| Metric | Unit | What it tells you |
|---|---|---|
DCGM_FI_PROF_SM_ACTIVE | % | True GPU utilization — fraction of time at least one warp is active on any SM. This is the "GPU utilization %" most teams want. |
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE | % | Tensor core utilization — critical for H100/B200 training workloads. Low means the model isn't leveraging the hardware's core capability. |
DCGM_FI_PROF_DRAM_ACTIVE | % | Memory-bandwidth utilization — high = memory-bound workload. Low SM + Low DRAM = idle GPU. |
DCGM_FI_DEV_GPU_UTIL | % | Coarse busy% (nvidia-smi equivalent) — useful for dashboards but less precise than SM_ACTIVE. |
DCGM_FI_DEV_MEM_COPY_UTIL | % | PCIe/DRAM copy-engine utilization — high = data-transfer bottleneck (host↔device). |
Health & reliability ("Is something wrong?")
| Metric | Unit | What it tells you |
|---|---|---|
DCGM_FI_DEV_XID_ERRORS | event | Last XID error code. XID 43, 63, 79 are especially bad — hardware/driver failures. |
DCGM_FI_DEV_ECC_SBE_VOL_TOTAL | count | Single-bit ECC errors — early warning of hardware degradation. |
DCGM_FI_DEV_ECC_DBE_VOL_TOTAL | count | Double-bit ECC errors — hardware failure; job likely corrupted. |
DCGM_FI_DEV_GPU_TEMP | °C | Die temperature. Approaching thermal throttle? For sustained full-load training, this is the first metric to alarm on. |
DCGM_FI_DEV_CLOCK_THROTTLE_REASONS | bitmask | Why is the GPU running slower? Thermal? Power cap? Software limit? |
Memory ("Will I OOM?")
| Metric | Unit | What it tells you |
|---|---|---|
DCGM_FI_DEV_FB_USED | MiB | Frame buffer (GPU memory) in use |
DCGM_FI_DEV_FB_FREE | MiB | Frame buffer free — approaching 0 = imminent OOM crash |
NVLink / interconnect ("Is communication the bottleneck?")
| Metric | Unit | What it tells you |
|---|---|---|
DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL | counter | Total NVLink bandwidth — for p5.48xlarge NVSwitch topology, low = model not using tensor parallelism effectively |
DCGM_FI_PROF_NVLINK_TX_BYTES / RX_BYTES | bytes/s | Per-link throughput — identifies asymmetric or saturated links |
Condensed implementation guide
Option A: Systemd on EC2 / ParallelCluster (training clusters)
Prerequisites:
- NVIDIA driver + DCGM library present on the AMI (standard on AWS Deep Learning AMIs)
- Amazon Managed Service for Prometheus (AMP) workspace created
- Head-node IAM role with
aps:RemoteWritepermission
Step 1 — Install DCGM Exporter on GPU compute nodes
Deploy via ParallelCluster's OnNodeConfigured hook or bake into the AMI:
#!/bin/bash # Only install on GPU nodes command -v nvidia-smi &>/dev/null || exit 0 DCGM_EXPORTER_VERSION="3.3.9-3.6.1" curl -fsSLo /tmp/dcgm-exporter.tar.gz \ "https://github.com/NVIDIA/dcgm-exporter/releases/download/${DCGM_EXPORTER_VERSION}/dcgm-exporter-${DCGM_EXPORTER_VERSION}-linux-amd64.tar.gz" mkdir -p /usr/local/dcgm-exporter tar -xzf /tmp/dcgm-exporter.tar.gz -C /usr/local/dcgm-exporter --strip-components=1 # Create systemd unit cat > /etc/systemd/system/dcgm-exporter.service <<'EOF' [Unit] Description=NVIDIA DCGM Exporter After=network-online.target ConditionPathExists=/usr/bin/nvidia-smi [Service] ExecStart=/usr/local/dcgm-exporter/dcgm-exporter \ --address=:9400 \ --collectors=/usr/local/dcgm-exporter/etc/dcp-metrics-included.csv \ --kubernetes=false Restart=on-failure [Install] WantedBy=multi-user.target EOF systemctl daemon-reload systemctl enable --now dcgm-exporter.service
Verify: curl -s http://localhost:9400/metrics | grep DCGM_FI_DEV_GPU_UTIL
Step 2 — Configure Prometheus agent on the head node
Run Prometheus in agent mode (scrape + forward, no local storage):
# /etc/prometheus/prometheus.yml global: scrape_interval: 15s external_labels: cluster: my-training-cluster scrape_configs: - job_name: dcgm-exporter ec2_sd_configs: - region: us-west-2 port: 9400 filters: - name: tag:parallelcluster:node-type values: ["Compute"] relabel_configs: - source_labels: [__meta_ec2_instance_id] target_label: instance_id - source_labels: [__meta_ec2_instance_state] regex: "(pending|running)" action: keep remote_write: - url: https://aps-workspaces.us-west-2.amazonaws.com/workspaces/<WORKSPACE_ID>/api/v1/remote_write sigv4: region: us-west-2 queue_config: max_samples_per_send: 5000 max_shards: 100
Start with --enable-feature=agent flag:
prometheus --config.file=/etc/prometheus/prometheus.yml --enable-feature=agent
Step 3 — Import the Grafana dashboard
In Amazon Managed Grafana: Dashboards → Import → grafana.com dashboard: 12239. This is the canonical NVIDIA DCGM Exporter dashboard — GPU overview, temperature, power, memory, ECC errors, NVLink, all pre-built.
Option B: DaemonSet on EKS (inference clusters)
helm repo add nvidia-dcgm https://nvidia.github.io/dcgm-exporter/helm-charts helm repo update helm install dcgm-exporter nvidia-dcgm/dcgm-exporter \ --namespace monitoring --create-namespace \ --set serviceMonitor.enabled=true
If using AWS Distro for OpenTelemetry (ADOT) as your collector, configure the Prometheus receiver to scrape the DCGM exporter service and remote-write to AMP. If using the Prometheus Operator, the ServiceMonitor created by the Helm chart handles discovery automatically.
Optional: Attribute GPU metrics to Slurm jobs
For HPC/training clusters, add a Slurm prolog script that writes the job ID to DCGM Exporter's job-mapping directory:
#!/bin/bash # /etc/slurm/prolog.d/dcgm-jobmap.sh JOBMAP_DIR=/var/run/dcgm-exporter/jobmap mkdir -p "$JOBMAP_DIR" for gpu in $(echo "$CUDA_VISIBLE_DEVICES" | tr ',' ' '); do echo "$SLURM_JOB_ID" > "$JOBMAP_DIR/$gpu" done
Then add --hpc-jobmapping-dir=/var/run/dcgm-exporter/jobmap to the DCGM Exporter startup command. Every metric gains an hpc_job label — enabling per-job queries like:
avg by (hpc_job)(DCGM_FI_PROF_SM_ACTIVE{cluster="my-cluster"})
Sizing and cost
For a typical GPU fleet:
| Component | Formula | Example (20 GPU instances × 8 GPUs each) |
|---|---|---|
| AMP ingestion | ~35 metrics/GPU × 4 samples/min × 160 GPUs × 43,200 min/mo | ~$140/mo |
| AMP storage | 150-day retention included | $0 additional |
| AMG | $9/active user/mo | $27–$90/mo (3–10 users) |
| DCGM Exporter (runtime) | Runs on existing GPU nodes | $0 incremental |
| Prometheus agent (head node) | Runs on existing head node in agent mode | $0 incremental |
| Total | ~$170–$230/mo for a 160-GPU fleet |
Compare this to the fleet it monitors: 20 × p5.48xlarge at ~$100/hr on-demand = $72,000/mo (or ~$30,000/mo after Savings Plans). The observability cost is 0.3–0.8% of the fleet cost — trivial, and it pays for itself the first time you catch a silent GPU waste event or a hardware failure before it corrupts a multi-day training run.
When NOT to use DCGM Exporter
- SageMaker Training Jobs / HyperPod: DCGM is already built in and configured. Don't install a separate exporter — use the native SageMaker observability stack. See Exported metrics reference.
- Single-GPU dev instances (g4dn.xlarge, etc.): The overhead of the monitoring stack likely exceeds the value for a single developer box. Use
nvidia-smimanually or the NVIDIA GPU monitoring CloudWatch integration available through certain AMIs. - Inference workloads on AWS Inferentia/Trainium chips: DCGM is NVIDIA-specific. For Inf2/Trn1 instances, use the Neuron Monitor and Neuron Prometheus plugin instead.
Well-Architected alignment
The AWS Well-Architected HPC Lens — Operate pillar explicitly calls out job-level observability as a best practice:
"Log job-level statistics, track anomalies and integrate your environment logging into a dashboard... If using AWS ParallelCluster with Slurm, leverage Slurm accounting with AWS ParallelCluster to log job-level statistics in an external database."
The pattern described in this article — DCGM Exporter + Slurm Exporter + AMP + AMG — is the reference implementation cited by the HPC Lens for this best practice.
For the broader Operational Excellence pillar of the standard Well-Architected Framework, this addresses:
- OPS 5 (Implement telemetry): Application-level telemetry for GPU workloads
- OPS 8 (Analyze workload health): Dashboard visibility into GPU fleet health, anomaly detection on ECC errors and thermal throttling
- OPS 9 (Respond to events): Alarming on XID errors, double-bit ECC, and thermal limits enables event-driven response
Suggested alarm set (start conservative)
| Condition | Severity | Action |
|---|---|---|
DCGM_FI_DEV_XID_ERRORS > 0 (any XID event) | Critical | Page on-call — potential hardware failure |
DCGM_FI_DEV_ECC_DBE_VOL_TOTAL > 0 (double-bit ECC) | Critical | Page + drain node — job data likely corrupted |
DCGM_FI_DEV_GPU_TEMP > 85°C sustained 5 min | Warning | Investigate cooling / workload distribution |
DCGM_FI_DEV_CLOCK_THROTTLE_REASONS == thermal for > 10 min | Warning | Node may be underperforming; check airflow/placement |
DCGM_FI_PROF_SM_ACTIVE < 20% for > 1 hr on a running job (requires hpc_job label) | Warning | Silent GPU waste — job is allocated but not computing |
References
- DCGM Exporter (upstream): github.com/NVIDIA/dcgm-exporter — source, releases, metrics CSV
- NVIDIA DCGM install documentation: Install DCGM Exporter — systemd / OCI / K8s
- AWS HPC Blog (Jun 2026): Monitoring AWS Parallel Computing Service — managed AMP + AMG pattern
- AWS Prescriptive Guidance: Grafana monitoring dashboard for ParallelCluster — self-hosted alternative
- AWS ParallelCluster observability repo: aws-samples/aws-hpc-recipes — observability_for_pcs
- SageMaker HyperPod metrics reference: Exported metrics reference — same DCGM metrics, pre-integrated
- EKS ML cluster setup: Set up EKS cluster for AI/ML workloads (CLI) — DaemonSet deployment of DCGM Exporter
- NVIDIA DCGM Grafana dashboard: grafana.com/dashboards/12239
- AWS Well-Architected HPC Lens — Operate: HPC Lens
- AMP EC2 remote-write setup: Set up metrics ingestion from EC2
- Slurm Job Exporter (community): guilbaults/slurm-job-exporter — Prometheus exporter for Slurm cgroup accounting + NVIDIA GPU stats per job
Licensing note
The DCGM Exporter itself is Apache 2.0. However, the Prometheus Slurm Exporter (cea-hpc/slurm_exporter) is GPLv3 — if your organization has commercial IP, run this through your OSS compliance review before adopting. The community slurm-job-exporter (guilbaults/slurm-job-exporter) is also GPLv3. An alternative is to use DCGM Exporter's built-in --hpc-jobmapping-dir with a simple prolog/epilog script (shown above) which avoids the GPLv3 dependency entirely.
Questions or feedback? Post in the comments. If you've deployed this pattern at scale and found additional metrics worth alarming on, share your experience — the community benefits from real-world operational learnings.
Relevant content
- AWS OFFICIALUpdated a year ago
asked 4 years ago
