Deploying GLM-5.2 on Amazon EKS using vLLM, Deep Learning Containers (DLCs), and FSx for Lustre
Example on how to deploy GLM-5.2 on Amazon EKS using vLLM, DLCs, P5en instances and FSx for Lustre
GLM-5.2 is one of the strongest open-weight large language models available today - a ~753B-parameter Mixture-of-Experts (MoE) model with DeepSeek-style sparse attention. Its FP8 checkpoint (zai-org/GLM-5.2-FP8) is roughly 705 GB across 141 safetensors shards, and it supports a context window of up to 1 million tokens.
In this post we deploy GLM-5.2-FP8 on Amazon EKS across 2x p5en.48xlarge instances (16× NVIDIA H200 GPUs total, ~2.26 TB of HBM3e). We’ll leverage the AWS Deep Learning Container (DLCs) that are pre-configured with necessary libraries and dependencies, including Elastic Fabric Adapter (EFA) drivers optimized for high-throughput, low-latency inter-node communications and GPUDirect RDMA support for running distributed inferencing.
We combine several building blocks that each solve a specific problem:
- vLLM on the AWS Deep Learning Container (DLC) as the inference engine - the stock vLLM 0.23.0 DLC already supports GLM-5.2. We'll just be adding a thin layer (Run:AI streamer + Ray) on top of it.
- Tensor parallelism (TP=8) inside each node over NVLink, and pipeline parallelism (PP=2) across the two nodes over Elastic Fabric Adapter (EFA) with GPUDirect RDMA.
- LeaderWorkerSet (LWS) and Ray to model and coordinate the two-node serving group on Kubernetes.
- Amazon FSx for Lustre as a shared, high-throughput file system for the model weights, paired with the Run:AI Model Streamer for fast loading.
- An EC2 Capacity Block for ML for 2x p5en.48xlarge instances
The model is sharded two ways at once: PP=2 splits the transformer layers across the two nodes, and
within each node TP=8 shards every layer across its 8 GPUs. A LeaderWorkerSet describes the group
(1 leader + 1 worker); Ray forms the two-node cluster, and vLLM runs on top with
--tensor-parallel-size 8 --pipeline-parallel-size 2.
This guide assumes you have intermediate Kubernetes experiences and are familiar with Amazon EKS and AWS CLI.
All artefacts used by this post are available here.
Prerequisites
- An EC2 Capacity Block for ML reservation for 2×
p5en.48xlarge. Note its reservation ID and Availability Zone - the FSx file system and the GPU node group both pin to that single AZ. eksctl≥ v0.205.0 (native Capacity-Block support), pluskubectl,helm, and the AWS CLI.- A Hugging Face token with access to
zai-org/GLM-5.2-FP8. - An x86 build host with Docker, and a private Amazon ECR repository in your region.
- Familiarity with EKS, and IAM permissions to create clusters, node groups, and FSx file systems.
- The following tools:
We use region ap-northeast-2 (Seoul) and Kubernetes 1.36 throughout this post, adjust these to your own environment.
Set the below in your shell:
export AWS_REGION="ap-northeast-2"; export REGION="$AWS_REGION" export ACCOUNT_ID="<ACCOUNT_ID>" export CLUSTER_NAME="glm52-poc-cluster" export K8S_VERSION="1.36" export ECR_REPO="glm52-vllm"; export IMAGE_TAG="glm52-fp8" export IMAGE_URI="${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com/${ECR_REPO}:${IMAGE_TAG}" export DLC_BASE="763104351884.dkr.ecr.${REGION}.amazonaws.com/vllm:0.23.0-gpu-py312-cu130-ubuntu22.04-ec2" export HF_MODEL="zai-org/GLM-5.2-FP8" export MODEL_DIR="/mnt/fsx/glm-5.2-fp8" export HF_TOKEN="<hf_token>" # rotate after use
Step 1 - Create an EKS cluster with CPU nodes only
We create the EKS control plane and a small CPU-based system node group first, with no GPU nodes. The GPU node group comes later, once the ML Capacity Block is active - this lets us build everything else (FSx, the customized vLLM image, the weights) ahead of the reservation window.
envsubst < 01-cluster/cluster.yaml | eksctl create cluster -f -
The cluster.yaml spans 2x Availability Zones (ap-northeast-2a + 2b), with the ML Capacity Block (for the 2x p5en.48xlarge) available in ap-northeast-2a.
The system node group uses privateNetworking: true so all nodes (CPU + GPU) and FSx stay in private subnets.
Cluster creation takes about 15 minutes.
Step 2 - Create an FSx for Lustre file system
FSx for Lustre gives all serving pods a shared, high-throughput ReadWriteMany volume for the weights.
On FSx for Lustre, capacity determines the number of Object Storage Targets (OSTs) - at the 1000 MB/s/TiB tier, 1 OST = 4.8 TiB. Weight-load time scales almost linearly with OST count (more on this in Results). We use 19.2 TiB = 4 OSTs as a balanced baseline.
First, create a dedicated security group. FSx validates its network settings at creation, and an EFA-enabled file system needs a self-referencing rule in both directions plus the Lustre ports:
export VPC_ID=$(aws eks describe-cluster --name "$CLUSTER_NAME" --region "$REGION" \ --query 'cluster.resourcesVpcConfig.vpcId' --output text) export NODE_SG_ID=$(aws eks describe-cluster --name "$CLUSTER_NAME" --region "$REGION" \ --query 'cluster.resourcesVpcConfig.clusterSecurityGroupId' --output text) export FSX_SG_ID=$(aws ec2 create-security-group --region "$REGION" \ --group-name glm52-fsx-lustre-sg --description 'GLM-5.2 FSx Lustre' --vpc-id "$VPC_ID" \ --query 'GroupId' --output text) aws ec2 authorize-security-group-ingress --group-id "$FSX_SG_ID" --protocol -1 --source-group "$FSX_SG_ID" --region "$REGION" aws ec2 authorize-security-group-egress --group-id "$FSX_SG_ID" --protocol -1 --source-group "$FSX_SG_ID" --region "$REGION" aws ec2 authorize-security-group-ingress --group-id "$FSX_SG_ID" --protocol tcp --port 988 --source-group "$NODE_SG_ID" --region "$REGION" aws ec2 authorize-security-group-ingress --group-id "$FSX_SG_ID" --protocol tcp --port 1018-1023 --source-group "$NODE_SG_ID" --region "$REGION"
Then create the file system in the Capacity Block's AZ (the FSx API takes a subnet ID, so we look up the private subnet in that AZ):
export FSX_CAPACITY_GIB="19200" export FSX_THROUGHPUT_TIER="1000" export CB_AZ="ap-northeast-2a" export CB_SUBNET_ID=$(aws ec2 describe-subnets --region "$REGION" \ --filters "Name=vpc-id,Values=$VPC_ID" "Name=availability-zone,Values=$CB_AZ" \ "Name=tag:kubernetes.io/role/internal-elb,Values=1" \ --query 'Subnets[0].SubnetId' --output text) echo "CB_SUBNET_ID=$CB_SUBNET_ID" aws fsx create-file-system --region "$REGION" \ --file-system-type LUSTRE --storage-type SSD \ --storage-capacity "$FSX_CAPACITY_GIB" \ --subnet-ids "$CB_SUBNET_ID" --security-group-ids "$FSX_SG_ID" \ --lustre-configuration '{ "DeploymentType":"PERSISTENT_2", "PerUnitStorageThroughput":'"$FSX_THROUGHPUT_TIER"', "EfaEnabled":true, "MetadataConfiguration":{"Mode":"AUTOMATIC"}, "DataCompressionType":"NONE" }'
Once it reaches AVAILABLE, install the FSx CSI driver and bind a static PV/PVC:
helm repo add aws-fsx-csi-driver https://kubernetes-sigs.github.io/aws-fsx-csi-driver/ && helm repo update helm upgrade --install aws-fsx-csi-driver aws-fsx-csi-driver/aws-fsx-csi-driver -n kube-system aws fsx describe-file-systems --region "$REGION" \ --query 'FileSystems[-1].[FileSystemId,LustreConfiguration.MountName,Lifecycle]' --output table export FSX_ID=<from the output above>; export FSX_MOUNT_NAME=<from the output above> export FSX_DNS="${FSX_ID}.fsx.${REGION}.amazonaws.com" envsubst < 02-fsx/fsx-pv.yaml | kubectl apply -f - envsubst < 02-fsx/fsx-pvc.yaml | kubectl apply -f - $ kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE fsx-glm52-pv 19200Gi RWX Retain Bound default/fsx-glm52-pvc <unset> 11s $ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE fsx-glm52-pvc Bound fsx-glm52-pv 19200Gi RWX <unset> 11s
Use a helper pod to confirm the mount and verify the four OSTs:
kubectl apply -f 02-fsx/fsx-test-pod.yaml kubectl wait --for=condition=Ready pod/fsx-test --timeout=180s $ kubectl exec -it fsx-test -- lfs df -h /mnt/fsx UUID bytes Used Available Use% Mounted on bok2lbev-MDT0000_UUID 549.9G 9.5M 549.9G 1% /mnt/fsx[MDT:0] bok2lbev-OST0000_UUID 4.5T 7.8M 4.5T 1% /mnt/fsx[OST:0] bok2lbev-OST0001_UUID 4.5T 7.5M 4.5T 1% /mnt/fsx[OST:1] bok2lbev-OST0002_UUID 4.5T 7.5M 4.5T 1% /mnt/fsx[OST:2] bok2lbev-OST0003_UUID 4.5T 7.5M 4.5T 1% /mnt/fsx[OST:3] filesystem_summary: 18.0T 30.2M 18.0T 1% /mnt/fsx
Step 3 - Add the Capacity-Block GPU node group
A single eksctl config consumes the Capacity Block and builds the launch template.
export CB_RESERVATION_ID="<CB_RESERVATION_ID>" export CB_AZ="ap-northeast-2a" eksctl create nodegroup -f <(envsubst < 03-nodegroup/nodegroup.yaml) # When the CB is active, scale the node group from 0 to 2: eksctl scale nodegroup --cluster "$CLUSTER_NAME" --name p5en-cb --nodes 2 --nodes-max 2 --region "$REGION"
Once the nodes join, install the NVIDIA device plugin and verify the GPUs are allocatable resources on each node:
kubectl apply -f 03-nodegroup/nvidia-device-plugin.yaml for NODE in $(kubectl get nodes -l role=gpu -o name); do echo -n "$NODE gpu="; kubectl get "$NODE" -o jsonpath='{.status.allocatable.nvidia\.com/gpu}' echo -n " efa="; kubectl get "$NODE" -o jsonpath='{.status.allocatable.vpc\.amazonaws\.com/efa}{"\n"}' done node/ip-192-168-72-123.ap-northeast-2.compute.internal gpu=8 efa=16 node/ip-192-168-88-156.ap-northeast-2.compute.internal gpu=8 efa=16
Step 4 - Validate the network fabric with a NCCL test
To prove the inter-node high-bandwidth, low-latency fabric works, we run the standard NCCL all_reduce_perf test across both p5en nodes using the public nccl-tests image.
# Install the Kubeflow MPI Operator once (provides the MPIJob CRD): kubectl apply --server-side -f https://raw.githubusercontent.com/kubeflow/mpi-operator/v0.8.0/deploy/v2beta1/mpi-operator.yaml kubectl get crd mpijobs.kubeflow.org kubectl apply -f 04-efa-validation/nccl-test-mpijob.yaml kubectl logs -f <nccl-tests-launcher-pod>
Look for NCCL INFO NET/OFI Selected Provider is efa (confirming EFA transport, not a TCP fallback):
[1,6]<stdout>:nccl-tests-worker-0:770:770 [6] NCCL INFO NET/OFI Plugin selected platform: AWS
[1,6]<stdout>:nccl-tests-worker-0:770:770 [6] NCCL INFO NET/OFI Configuring AWS-specific options
[1,6]<stdout>:nccl-tests-worker-0:770:770 [6] NCCL INFO NET/OFI Internode latency set at 35.0 us
[1,6]<stdout>:nccl-tests-worker-0:770:770 [6] NCCL INFO NET/OFI Using transport protocol RDMA (platform set)
[1,12]<stdout>:nccl-tests-worker-1:764:764 [4] NCCL INFO NET/OFI Selected provider is efa, fabric is efa-direct (found 16 nics)
On our 2-node, 16-GPU setup, bus bandwidth ramped to ~488 GB/s at the large-message tail - combining intra-node NVSwitch and cross-node EFA GPUDirect RDMA. (look for a busbw column that climbs to a plateau at the large-message sizes, and 0 wrong at every size. )
[1,0]<stdout>:# out-of-place in-place
[1,0]<stdout>:# size count type redop root time algbw busbw #wrong time algbw busbw #wrong
[1,0]<stdout>:# (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)
[truncated]
[1,0]<stdout>: 536870912 134217728 float sum -1 2466.66 217.65 408.10 0 2456.98 218.51 409.70 0
[1,0]<stdout>: 1073741824 268435456 float sum -1 4531.65 236.94 444.27 0 4537.00 236.66 443.74 0
[1,0]<stdout>: 2147483648 536870912 float sum -1 9000.57 238.59 447.36 0 9034.22 237.71 445.70 0
[1,0]<stdout>: 4294967296 1073741824 float sum -1 17349.7 247.55 464.16 0 17369.6 247.27 463.63 0
[1,0]<stdout>: 8589934592 2147483648 float sum -1 33684.6 255.01 478.15 0 33685.6 255.00 478.13 0
[1,0]<stdout>: 17179869184 4294967296 float sum -1 66054.7 260.09 487.66 0 66006.7 260.27 488.01 0
Step 5 - Build the custom vLLM image
The stock vLLM 0.23.0 DLC already supports GLM-5.2's GlmMoeDsaForCausalLM architecture, so we don't need to upgrade vLLM or torch.
Our image is a thin layer that adds exactly two things:
- The Run:AI Model Streamer, which enables
--load-format runai_streamerfor concurrent weight loading. - Ray, which is required for multi-node vLLM serving.
aws ecr get-login-password --region "$REGION" | docker login --username AWS --password-stdin 763104351884.dkr.ecr.${REGION}.amazonaws.com aws ecr get-login-password --region "$REGION" | docker login --username AWS --password-stdin ${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com aws ecr create-repository --repository-name "$ECR_REPO" --region "$REGION" 2>/dev/null || true docker build --platform linux/amd64 --build-arg DLC_BASE="$DLC_BASE" -t "$IMAGE_URI" 05-image/ docker push "$IMAGE_URI"
Step 6 - Stage the weights onto FSx
A one-time Kubernetes Job downloads the ~705 GB model into the FSx volume using the Hugging Face Xet high-performance backend:
kubectl create secret generic hf-token --from-literal=token="$HF_TOKEN" envsubst '${MODEL_DIR} ${HF_MODEL}' < 06-weights/download-weights-job.yaml | kubectl apply -f - $ kubectl get job glm52-weight-download -w NAME STATUS COMPLETIONS DURATION AGE glm52-weight-download Running 0/1 19s 19s glm52-weight-download Running 0/1 13m 13m glm52-weight-download SuccessCriteriaMet 0/1 14m 14m glm52-weight-download Complete 1/1 14m 14m
Once it completes, confirm the weights spread evenly across all four OSTs (approx. 176G per OST):
$ kubectl exec fsx-test -- lfs df -h /mnt/fsx UUID bytes Used Available Use% Mounted on bok2lbev-MDT0000_UUID 549.9G 12.8M 549.9G 1% /mnt/fsx[MDT:0] bok2lbev-OST0000_UUID 4.5T 178.3G 4.3T 4% /mnt/fsx[OST:0] bok2lbev-OST0001_UUID 4.5T 173.2G 4.3T 4% /mnt/fsx[OST:1] bok2lbev-OST0002_UUID 4.5T 176.2G 4.3T 4% /mnt/fsx[OST:2] bok2lbev-OST0003_UUID 4.5T 176.4G 4.3T 4% /mnt/fsx[OST:3] filesystem_summary: 18.0T 704.1G 17.3T 4% /mnt/fsx
Step 7 - Deploy GLM-5.2 with LeaderWorkerSet and Ray
Install the LeaderWorkerSet controller, then deploy the two-node serving group and a ClusterIP Service:
helm install lws oci://registry.k8s.io/lws/charts/lws --version 0.9.0 -n lws-system --create-namespace --wait kubectl -n lws-system get pods # wait for lws-controller-manager to be Running before applying below envsubst '$IMAGE_URI $MODEL_DIR' < 07-serving/lws-serving.yaml | kubectl apply -f - kubectl apply -f 07-serving/service.yaml kubectl get pods -l app=glm52 -w # glm52-0 (leader) + glm52-0-1 (worker); wait for glm52-0 = 1/1
The leader starts the Ray head, waits for the worker to register all 16 GPUs, then launches vLLM:
vllm serve /mnt/fsx/glm-5.2-fp8 \ --served-model-name glm-5.2-fp8 \ --tensor-parallel-size 8 --pipeline-parallel-size 2 \ --distributed-executor-backend ray \ --load-format runai_streamer \ --model-loader-extra-config '{"concurrency":32,"memory_limit":10737418240}' \ --kv-cache-dtype fp8 \ --max-model-len 1048576 \ --gpu-memory-utilization 0.90 \ --trust-remote-code \ --tool-call-parser glm47 --enable-auto-tool-choice \ --host 0.0.0.0 --port 8000
A few things worth calling out in the serving manifest:
- We set TP=8 for intra-node parallelism over NVLink, and PP=2 for inter-node pipeline parallelism using GPUDirect RDMA (over EFA)
- We set
--max-model-len 1048576to provide GLM-5.2's full 1M-context window. - We point
VLLM_CACHE_ROOTat the FSx volume. This persists vLLM'storch.compileand DeepGEMM kernel caches on shared storage for warm start. - We run the Run:AI streamer at concurrency 32
Test it
From any in-cluster pod (or use the fsx test pod), hit the OpenAI-compatible endpoint:
kubectl exec fsx-test -- curl -sS http://glm52-leader.default.svc.cluster.local:8000/v1/chat/completions \ -H content-type:application/json \ -d '{"model":"glm-5.2-fp8","messages":[{"role":"user","content":"What is Kubernetes? Answer in 3 sentences."}],"max_tokens":300}' \ | python3 -m json.tool
GLM-5.2 is a reasoning model - it "thinks out loud" before giving its answer, so expect a verbose trace.
Benchmark
Throughput tests (2 nodes, vllm bench serve, 200 prompts, concurrency 64, 1024-in / 256-out, run from the LWS LEADER pod):
$ kubectl exec glm52-0 -- bash -lc 'vllm bench serve --backend openai-chat --model /mnt/fsx/glm-5.2-fp8 \ --served-model-name glm-5.2-fp8 --tokenizer /mnt/fsx/glm-5.2-fp8 \ --base-url http://localhost:8000 --endpoint /v1/chat/completions \ --dataset-name random --num-prompts 200 --max-concurrency 64 \ --random-input-len 1024 --random-output-len 256'
| Metric | Value |
|---|---|
| Total token throughput | ~7,900 tok/s |
| Output token throughput | ~1,570 tok/s (peak ~2,000) |
| Median time-to-first-token | ~274 ms |
| Median inter-token latency | ~35 ms |
Storage and cold start. We measured the weight-load phase across different FSx OST counts (loading the same 705 GB, dropping the OS page cache before each run):
| FSx OSTs | Capacity | Avg weight-load | Full cold start (warm caches) |
|---|---|---|---|
| 2 | 9.6 TiB | ~676 s | ~840 s |
| 4 | 19.2 TiB | ~382 s | ~540 s |
| 6 | 28.8 TiB | ~274 s | ~400 s |
Cleanup
To avoid incurring long-term charges, delete the AWS resources created as part of the demo walkthrough.
kubectl delete lws glm52 ; kubectl delete svc glm52-leader kubectl delete job glm52-weight-download ; kubectl delete secret hf-token kubectl delete pvc fsx-glm52-pvc ; kubectl delete pv fsx-glm52-pv aws fsx delete-file-system --file-system-id "$FSX_ID" --region "$REGION" eksctl delete cluster --name "$CLUSTER_NAME" --region "$REGION"
Conclusion
In this post, we walked through a deployment example for running the GLM-5.2 model on Amazon EKS using vLLM, DLCs, served by EC2 p5en.48xlarge instances with FSx for Lustre integrations.
References
Relevant content
- AWS OFFICIALUpdated 2 years ago
