Resolving EKS Pod IP exhaustion caused by VPC CNI warm pool defaults

Introduction

Amazon Elastic Kubernetes Service (Amazon EKS) Pods fail with "FailedCreatePodSandBox" or "InsufficientFreeAddressesInSubnet" errors even though subnets appear to have available IP addresses. The default Amazon VPC container network interface (CNI) plugin warm pool behavior is the root cause. Without notification, the warm pool reserves IP addresses at the elastic network interface level. Pod IP address assignment failures in Amazon Elastic Kubernetes Service (Amazon EKS) clusters are among the most difficult networking issues to diagnose. The VPC CNI warm pool silently reserves too many IP addresses without obvious symptoms and triggers an alarm only after Pod scheduling fails.

Solution overview

This article describes a real-world scenario where a large enterprise experienced repeated subnet exhaustion across multiple clusters. It provides the diagnostic steps and resolution that their Technical Account Manager (TAM) and AWS Support engineers implemented to address the issue. You'll learn how to diagnose the failure pattern and apply the correct VPC CNI configuration to stop excessive IP address reservation. You'll also implement enhanced subnet discovery to provide fresh IP address space without disrupting existing workloads.

Customer scenario

A large enterprise that runs over 300 microservices on Amazon EKS in a hub-and-spoke architecture with AWS Transit Gateway began to experience intermittent Pod scheduling failures across multiple clusters. Initially, the platform team added secondary CIDR blocks and created additional subnets every 2–3 months to extend the VPCs. They repeated this pattern several times but didn’t understand why subnets continued to fill. The issue escalated to critical priority when a production deployment failed because no IP addresses were available. The platform team then engaged their TAM for root cause analysis.

During the initial consultation, the TAM and an EKS Support subject matter expert (SME) reviewed the cluster's VPC CNI configuration. They found that the default warm pool setting, a WARM_ENI_TARGET value of 1, reserved approximately 30 IP addresses for each node regardless of the Pod count. With 10-15 nodes sharing /24 subnets, the warm pool used most of the available addresses.

The TAM then collaborated with AWS Support engineers to validate the fix across the platform team's environment with multiple AWS accounts. The TAM and Support engineers confirmed that service-side constraints didn’t exist and developed a phased rollout plan.

The team explored secondary CIDR blocks and overlay networking as longer-term options. However, the immediate fix required no architectural changes. It only required tuning the VPC CNI warm pool parameters so that it no longer reserved too many IP addresses.

The team configured the following optimized environment values:

• A WARM_ENI_TARGET value of 0

• A WARM_IP_TARGET value of 5

• A MINIMUM_IP_TARGET value of 10

The cluster recovered previously locked IP addresses within minutes and has not experienced subnet exhaustion since.

The team combined the preceding settings with enhanced subnet discovery, which allows the VPC CNI to use additional tagged subnets for Pod IP address allocation. For more information, see Amazon VPC CNI introduces enhanced subnet discovery.

Note: Enhanced subnet discovery is activated by default in VPC CNI v1.18.0 and later.

Solution implementation

The Amazon VPC CNI plugin uses a warm pool to pre-allocate IP addresses on each node for fast Pod scheduling. The default configuration, WARM_ENI_TARGET=1, reserves one full network interface worth of IP addresses for each node, regardless of the Pod count. That's approximately 15-50 IP addresses depending on the instance type. In clusters with undersized subnets of /24 or /25, this default causes a gradual, undetected depletion of available subnet IP addresses that eventually results in Pod scheduling failures.

For more information about the warm pool settings, see eni-vpc-cni-K8s on the GitHub website.

Prerequisites

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

To implement this solution, you must use the following resources and tools:

• AWS CLI that's configured with the necessary permissions.

• kubectl that's configured to communicate with your EKS cluster.

• AWS Identity and Access Management (IAM) permissions, including the AmazonEKS_CNI_Policy AWS managed policy that you attach to the VPC CNI service account role.

• An existing Amazon EKS cluster with the Amazon VPC CNI add-on plugin version 1.18.0 or later installed. When you provision a cluster, Amazon EKS automatically installs VPC CNI. For more information, see Deploy VPC CNI Managed Add-On.

If your cluster uses an earlier version of the Amazon VPC CNI, then run the following update-addon command to update it:

aws eks update-addon --cluster-name EXAMPLE_CLUSTER_NAME --addon-name vpc-cni --resolve-conflicts OVERWRITE --region EXAMPLE_REGION

Note: Replace EXAMPLE_CLUSTER_NAME with the name of your cluster and EXAMPLE_REGION with your AWS Region.

Understanding the root cause

When the WARM_ENI_TARGET value is 1, each node reserves one extra network interface worth of IP addresses in its warm pool in addition to the IP addresses that its Pods use.

For example, you have a cluster of 10 nodes that use m5.xlarge instances with four network interfaces, and each network interface contains 15 IP addresses. In this cluster, each node holds about 30 IP addresses, or two network interfaces, even when the cluster runs only a few Pods. Approximately 15 IP addresses for each node, which is about 150 across the cluster, sit idle as warm-pool reservation. On a /24 subnet with 251 usable IP addresses, the warm-pool reservation uses more than half the subnet and keeps about 101 IP addresses for Pod workloads.

For information about network specifications for instances, see Network specifications.

Also, while Kubernetes controllers create and terminate Pods over weeks and months, the remaining free IP addresses randomly scatter across the subnet address space through normal application deployments and scaling.

This process, known as IP address fragmentation, creates two cascading failures:

• Individual IP address allocation fails because the subnet reaches zero available addresses.

• If you activated Prefix delegation, then it fails immediately. Prefix delegation requires 16 contiguous free IP addresses, or /28 block. These blocks are nearly impossible to find in heavily used, fragmented subnets.

This failure mode can be confusing because existing Pods continue to run normally. Alarms trigger only after a new Pod, node, or network interface attachment request fails.

Teams often respond by adding more nodes or requesting additional small subnets, but both responses accelerate exhaustion rather than resolve it. When you add more nodes, more warm pool reservations use even more IP addresses from the already-constrained subnet. Adding small /24 subnets provides temporary relief but creates the same exhaustion pattern within weeks.

Important: Activating prefix delegation on an exhausted or fragmented subnet can worsen the issue. The VPC CNI switches from assigning individual scattered IP addresses to demanding /28 contiguous blocks. When no contiguous blocks exist, all new Pod IP address requests fail.

Identifying the problem

To check the number of available IP addresses in your Pod subnets, run the following describe-subnets command:

aws ec2 describe-subnets --filters "Name=tag:kubernetes.io/role/cni,Values=1" --query "Subnets\[\].{Name:Tags\[?Key=='Name'\].Value\|\[0\],CIDR:CidrBlock,AvailableIPs:AvailableIpAddressCount}" --output table

Subnets with fewer than 10 available IP addresses or utilization above 80% are candidates for warm pool optimization.

To check your current VPC CNI warm pool configuration, run the following describe command:

kubectl describe ds aws-node -n kube-system \| grep -E "WARM_ENI\|WARM_IP\|MINIMUM_IP\|ENABLE_SUBNET"

If the WARM_ENI_TARGET variable is set to 1, or no warm pool variables are explicitly configured, then the default is active and contributes to IP address exhaustion.

Check if your current subnets have enough free space to support prefix delegation. If subnets are above 80% utilization with scattered free IP addresses, then prefix delegation fails and you see the following error in the VPC CNI logs:

“failed to allocate a private IP/Prefix address: InsufficientCidrBlocks: There are not enough free cidr blocks in the specified subnet to satisfy the request.”

To check your VPC CNI logs, see How do I resolve Amazon VPC CNI plugin issues for Amazon EKS?

Important: Activate prefix delegation only after fresh, unfragmented IP address space is available.

To check the current network interface and IP address allocation for each node, run the following describe-network-interfaces command:

aws ec2 describe-network-interfaces --filters "Name=description,Values=aws-K8S-*" --query "NetworkInterfaces\[\].{NodeIP:PrivateIpAddress,SubnetId:SubnetId,SecondaryIPs:PrivateIpAddresses\|length(@)}" --output table

The output shows the total number of IP addresses that each node holds compared with the number of IP addresses that are assigned to running Pods. The difference between these values represents the number of reserved IP addresses that workloads aren’t actively using, known as warm pool waste.

Tuning the VPC CNI warm pool

To update the VPC CNI add-on configuration and reduce warm pool waste, run the following update-addon command:

aws eks update-addon --cluster-name EXAMPLE_CLUSTER_NAME --addon-name vpc-cni --configuration-values '{"env":{"**WARM_ENI_TARGET**":"0","**WARM_IP_TARGET**":"5","**MINIMUM_IP_TARGET**":"10","ENABLE_SUBNET_DISCOVERY":"true"}}' --region EXAMPLE_REGION

Note: Replace EXAMPLE_CLUSTER_NAME with the name of your cluster and EXAMPLE_REGION with your Region.

The preceding environment value configurations have the following effects:

• A WARM_ENI_TARGET value of 0 deactivates the reservation of a full network interface for each node. This is the primary cause of IP address waste in default configurations.

• A WARM_IP_TARGET value of 5 keeps only five extra IP addresses for each node for fast Pod scheduling. This value is sufficient for typical workloads without excessive reservation.

• A MINIMUM_IP_TARGET value of 10 sets the minimum number of IP addresses so that at least 10 IP addresses are pre-allocated on each node. It provides a buffer for sudden Pod scheduling bursts without throttling Amazon Elastic Compute Cloud (Amazon EC2) API calls.\

  Note: MINIMUM_IP_TARGET sets the base and takes precedence over WARM_IP_TARGET when you configure both settings.

• ENABLE_SUBNET_DISCOVERY=true confirms that enhanced subnet discovery is active.\

  Important: Use the aws eks update-addon command rather than kubectl set env. The update-addon command keeps the configuration after add-on updates. Settings that you apply through kubectl set env reset to defaults when you update the VPC CNI add-on. Your warm pool optimization resets, Amazon EKS doesn’t notify you, and the exhaustion pattern returns.

After you update the VPC CNI add-on configuration, the aws-node DaemonSet Pods perform a rolling restart. However, your running Pods don't restart and only new IP address allocations use the updated settings. Soon after you update the configuration, previously reserved warm pool IP addresses are freed back to the subnet.

Adding IP address space with enhanced subnet discovery

If warm pool tuning doesn't free enough IP addresses, such as when subnets already have zero available addresses, then create new larger subnets. You must tag the new subnets so that the VPC CNI automatically discovers them.

To create larger subnets, complete the following steps:

1. Create new subnets that are sized /22 or larger, with 1,019+ usable IP addresses in each Availability Zone that your cluster uses. If your primary VPC CIDR is constrained, then use the 100.64.0.0/10 CIDR range for CG-NAT space. The 100.64.0.0/10 CIDR range doesn't use RFC 1918 space that you might need for other workloads or on-premises routing.

2. Run the following create-tags command to tag the new subnets for VPC CNI discovery:
   aws ec2 create-tags --resources EXAMPLE_SUBNET_ID --tags Key=kubernetes.io/role/cni,Value=1 --region EXAMPLE_REGION
   Note: Replace EXAMPLE_SUBNET_ID with the ID of your subnet, and EXAMPLE_REGION with your Region.

3. Check that you associated the new subnets with the correct route tables and network access control lists (network ACLs) for Pod traffic. Pods require NAT gateway access for internet-bound traffic and connectivity to VPC endpoints for AWS service access.

4. Check that the IAM role that you attached to the VPC CNI service account contains the ec2:DescribeSubnets permission. This permission is included in the AmazonEKS_CNI_Policy AWS managed policy by default.

The VPC CNI automatically discovers subnets that you tagged with kubernetes.io/role/cni=1 and begins to allocate Pod IP addresses without requiring a node restart. Existing Pods continue to use their current IP addresses without disruption. New Pods receive IP addresses from the fresh, unfragmented subnets.

Note: You don't need to remove or modify existing subnets. The old subnets remain as-is for existing Pods. The VPC CNI preferentially allocates from subnets with more available IP address space. To explicitly exclude a subnet from VPC CNI discovery, tag it with kubernetes.io/role/cni=0.

For production workloads with higher Pod density and scaling requirements, use /20 subnets each with 4,091 usable IP addresses for each Availability Zone. These subnets provide additional space so that you don't reach the same exhaustion pattern when clusters scale.

When you apply warm pool tuning and enhanced subnet discovery together, clusters immediately recover previously locked IP addresses. The clusters also gain fresh IP address space without disrupting existing workloads.

Validating the fix

To confirm that you resolved the issue, complete the following steps:

1. Run the following get daemonset command to check that warm pool settings are applied:
   kubectl get daemonset aws-node -n kube-system -o jsonpath='{.spec.template.spec.containers\[0\].env}' \| python3 -m json.tool

2. Confirm that the ENABLE_SUBNET_DISCOVERY, WARM_IP_TARGET, MINIMUM_IP_TARGET, and WARM_ENI_TARGET settings reflect your changes.

3. Run the following get Pods command to confirm that no Pods are stuck in Pending or Error state:
   kubectl get Pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded

4. Run the following get Pods command to confirm that new Pods receive IP addresses from the expected subnets:
   kubectl get Pods -A -o wide

5. Confirm that new Pod IP addresses are within the CIDR range of the newly created subnets.

6. Run the following describe-subnets command to monitor subnet IP address recovery for 24-48 hours:
   aws ec2 describe-subnets --subnet-ids EXAMPLE_SUBNET_ID --query "Subnets\[0\].AvailableIpAddressCount"
   Note: Replace EXAMPLE_SUBNET_ID with the ID of your subnet.

While the updated warm pool settings take effect across all nodes, expect a significant increase in available IP addresses.

Best practices to prevent recurrence

To prevent the issue from recurring, implement the following best practices:

• Size new Pod subnets at /22 or larger for clusters with 10 or more nodes. Undersized /24 or /25 subnets primarily cause this failure pattern.

• Deploy the CNI metrics helper to monitor IP address utilization at the Pod networking level.

• Use VPC IP Address Manager (IPAM) with Amazon CloudWatch to monitor at the infrastructure level. Configure CloudWatch alarms on the SubnetIPAllocated metric at 80% utilization to provide early warning before exhaustion at the VPC layer.

• Incorporate optimized CNI settings into cluster provisioning templates, such as Terraform or CloudFormation, so that all new EKS clusters inherit the configuration by default.

• After fresh, unfragmented subnets are available, use prefix delegation as a follow-up optimization. Prefix delegation requires contiguous /28 blocks that exist only in healthy subnets.

• Review node scaling policies. Clusters that respond to Pod pressure by adding nodes rather than optimizing IP address usage, create a feedback loop. Each new node reserves its own warm pool IP addresses, which accelerates subnet exhaustion.

Conclusion

This article explained how the default VPC CNI warm pool configuration exhausts subnet IP addresses in Amazon EKS clusters without visible warning. It explained why prefix delegation worsens the issue on fragmented subnets. It also showed how to resolve the issue by using warm pool optimization combined with enhanced subnet discovery. By applying these settings through the EKS add-on AWS CLI command, the configuration remains across updates and prevents recurrence.

To learn more about how AWS Support can help you optimize your Amazon EKS environments, see AWS Enterprise Support.

Related information

Optimizing IP address utilization

Configure subnet selection for Pod IP addresses

Assign more IP addresses to Amazon EKS nodes with prefixes

Assign IP addresses to Pods with the Amazon VPC CNI

About the authors

Tejas Majamudar is a Technical Account Manager at AWS Enterprise Support, working with Global System Integrators (GSIs) in India. He specializes in Networking, Cloud Financial Management, AIOps, Resiliency, Container Orchestration, and Migration. He helps enterprise customers build resilient, cost-efficient infrastructure on AWS and focuses on turning complex operational challenges into reusable best practices for the broader community.

Karthik Veeram Reddygari is a Cloud Support Engineer at AWS specializing in Amazon EKS and Amazon ECS. He works with Enterprise Support customers to troubleshoot complex container networking issues and optimize their containerized environments.