Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

Running Blackwell-class GPU HPC on AWS Parallel Computing Service (PCS) with P6e-GB200 UltraServers and Slurm

4 minute read
Content level: Intermediate
0

This pattern is a direct fit for semiconductor and high-tech HPC: EDA verification, physical/logic simulation, and large-scale AI training all run on Slurm and increasingly need Blackwell-class GPUs. AWS PCS gives EDA/HPC teams the familiar Slurm experience without operating the controller, while P6e-GB200 UltraServers supply rack-scale GPU capacity.

Overview

For large-scale training, fine-tuning, and HPC simulation, teams increasingly need Blackwell-class GPU capacity behind a familiar Slurm scheduler — without operating the control plane themselves. AWS Parallel Computing Service (PCS) provisions a managed Slurm controller, operates the scaling logic, and launches compute nodes for you; you simply define job queues and compute preferences. (AWS HPC FAQs)

Pairing PCS with Amazon EC2 P6e-GB200 UltraServers (NVIDIA GB200 NVL72) gives you rack-scale GPU capacity under Slurm while AWS manages fleet scaling in a highly available configuration.

The building blocks

AWS PCS + Slurm

Without PCS you would run a Slurm controller on a head node, launch compute nodes, and manage fleet scaling yourself. PCS "is built to manage the Slurm controller and handles fleet scaling in a highly available and secure configuration." (AWS HPC FAQs) You organize capacity into compute node groups (CNGs) and can apply custom Slurm settings per CNG — for example Features, RealMemory, Sockets, and Weight. (PCS docs: Custom Slurm settings for compute node groups)

Tagging GPU/NVMe features lets users request them with standard Slurm constraints:

aws pcs update-compute-node-group \
  --cluster-identifier my-cluster \
  --compute-node-group-identifier my-cng-1 \
  --slurm-configuration 'SlurmCustomSettings=[{parameterName=Features,parameterValue="gpu,nvme"}]'

Users then submit with --constraint=gpu&nvme, just as on any other Slurm cluster. (AWS HPC blog: Custom Slurm Settings in AWS PCS)

P6e-GB200 UltraServers

P6e-GB200 UltraServers are accelerated by NVIDIA GB200 NVL72, connecting 36 NVIDIA Grace CPUs and 72 Blackwell GPUs in a single NVLink domain. They come in two sizes: u-p6e-gb200x36 (36 GPUs across 9 compute nodes) and u-p6e-gb200x72 (72 GPUs across 18 compute nodes). (AWS ML blog: P6e-GB200 UltraServers)

EC2 Capacity Blocks for ML

P6e-GB200 UltraServers are available only through EC2 Capacity Blocks for ML — you reserve the UltraServer (not individual instances). (Containers blog) Capacity Blocks place instances close together in EC2 UltraClusters, can be reserved up to eight weeks in advance, support 1–64 instances per block (up to 256 across blocks), and can be shared across multiple accounts. (EC2 Capacity Blocks for ML; EC2 User Guide: Capacity Blocks for ML)

Architecture at a glance

p6e-ultraservers-slurm

Diagram summarizes facts cited above (UltraServer sizes, Capacity Blocks reservation model, PCS managed Slurm controller, Features-based scheduling).

Reference workflow

  1. Reserve UltraServer capacity. In the EC2 console → Capacity Reservations → Capacity Blocks, choose the UltraServers tab and reserve u-p6e-gb200x36 or u-p6e-gb200x72 for your window. (Containers blog)
  2. (Multi-account) Group the reservation. For UltraServer Capacity Blocks, create a Capacity Reservation Resource Group so PCS/EC2 can target the reserved capacity — see "Create a Capacity Reservation Resource Group for UltraServer Capacity Blocks" in the EC2 User Guide. (EC2 User Guide)
  3. Create the PCS cluster and a GPU compute node group that launches into the reserved Capacity Block, tagging GPU/NVMe Features so jobs can constrain to them.
  4. Define queues mapping to the CNG, and submit Slurm jobs with --constraint=... to land on the right nodes.

Best practices

  • Reserve early and right-size the block. Capacity Blocks can be booked up to 8 weeks ahead; align the reservation window to your training/simulation schedule to avoid paying for idle GPU time.
  • Use a Capacity Reservation Resource Group for multi-account teams. It lets shared Blackwell capacity be targeted cleanly across accounts (Capacity Blocks are shareable across accounts).
  • Keep the NVLink domain intact for tightly-coupled jobs. UltraServers connect 72 GPUs in one NVLink domain — schedule tightly-coupled training to stay within an UltraServer for best interconnect performance.
  • Encode node capabilities as Slurm Features. Explicit Features (e.g. gpu, nvme) plus Weight let Slurm place jobs on the cheapest node that satisfies the constraint, reducing user error.
  • Plan for impairment handling. For UltraServer Capacity Blocks, AWS notifies you of detected impairment but generally will not terminate your instances automatically; you can request remediation via a support case. (EC2 Capacity Blocks pricing page)

Resources

No comments

Relevant content