Preventing pod and node disruption in Amazon EKS Auto Mode

When you run fault-intolerant workloads on Amazon Elastic Kubernetes Service (Amazon EKS) Auto Mode, you might need to extend the lifecycle of your nodes and pods beyond the default disruption policies. This is particularly relevant when you're managing application upgrades on your own timeline, rather than being forced into immediate updates when new Amazon Machine Images (AMIs) become available.

In this post, I'll show you how to configure disruption budgets in Amazon EKS Auto Mode to maximize node and pod lifetimes while staying within AWS's mandatory 21-day update window. I also demonstrate how to monitor these resources using Amazon CloudWatch to track when disruptions occur.

The challenge: balancing updates with operational needs

Amazon EKS Auto Mode automatically manage your cluster's compute resources, including node lifecycle management. By default, these systems optimize for cost and efficiency by consolidating underutilized nodes and replacing them when new AMIs are available. However, some workloads require more control over when disruptions occur.

Consider this scenario in EKS Auto Mode: AWS releases a new AMI with security patches, and you want to upgrade your application to be compatible with the new AMI within 20 days. You need your nodes to remain stable during this period so you can test and deploy your application updates without unexpected disruptions. After 21 days, AWS enforces mandatory node replacement for security compliance.

The question becomes: how do you prevent voluntary disruptions while respecting mandatory update requirements?

Understanding disruption types in EKS Auto Mode

Amazon EKS Auto Mode uses Karpenter under the hood to manage node lifecycle. Karpenter can disrupt nodes for several reasons:

• Drift: When a node's configuration no longer matches the desired state (such as a new AMI being available)
• Consolidation: When nodes are underutilized and can be replaced with smaller or fewer instances
• Expiration: When nodes exceed their configured maximum lifetime
• Emptiness: When nodes have no running pods

Each of these disruption methods can be controlled through NodePool disruption budgets and Pod Disruption Budgets (PDBs).

Solution: configuring NodePool disruption budgets

The most effective approach to preventing voluntary disruptions is to configure your NodePool with a disruption budget that blocks all voluntary disruptions. Here's how to set this up:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: prevent-disruptions-for-the-longest
spec:
  disruption:
    budgets:
    - nodes: "0%"
    consolidateAfter: 8760h
    consolidationPolicy: WhenEmptyOrUnderutilized
  template:
    spec:
      expireAfter: 480h
      terminationGracePeriod: 24h
      requirements:
      - key: kubernetes.io/arch
        operator: In
        values: ["amd64"]
      - key: karpenter.sh/capacity-type
        operator: In
        values: ["on-demand"]
      nodeClassRef:
        group: eks.amazonaws.com
        kind: NodeClass
        name: default

Let me break down the key configuration options:

Disruption budget: Setting nodes: "0%" blocks all voluntary disruptions. This prevents Karpenter from consolidating or replacing nodes due to drift or underutilization.

Consolidation policy: The consolidateAfter: 8760h (365 days) setting effectively disables time-based consolidation, while consolidationPolicy: WhenEmptyOrUnderutilized defines when consolidation might occur if the budget allowed it.

Expiration: The expireAfter: 480h (20 days) sets the maximum node lifetime. However, this is an upper bound, not a guarantee. Nodes can still be disrupted earlier if other disruption methods are allowed by the budget.

Termination grace period: The terminationGracePeriod: 24h gives pods 24 hours to gracefully shut down when a node must be terminated.

An important consideration: the expireAfter field defines the upper bound of node lifetime and it's not guaranteed, because nodes can be disrupted earlier by other disruption methods such as Consolidation. Hence for this use case, we set nodes: "0%" to block all voluntary disruptions. To enforce a true maximum lifetime, you must combine expireAfter with carefully configured disruption budgets that block other disruption reasons. See the post on Common Karpenter configuration pitfalls in Amazon EKS to learn more.

Verifying disruption prevention

After creating your NodePool configuration and deploying some Kubernetes workloads in the cluster, you can verify that disruptions are being blocked by checking the NodePool events:

kubectl describe nodepool prevent-disruptions-for-the-longest

You might see events like this:

Events:
  Type    Reason             Age                  From                   Message
  ----    ------             ----                 ----                   -------
  Normal  DisruptionBlocked  57m (x982 over 17h)  eks-auto-mode/compute  No allowed disruptions for disruption reason Underutilized due to blocking budget
  Normal  DisruptionBlocked  40m (x13 over 52m)   eks-auto-mode/compute  No allowed disruptions for disruption reason Underutilized due to blocking budget
  Normal  DisruptionBlocked  3m6s (x51 over 53m)  eks-auto-mode/compute  No allowed disruptions for disruption reason Drifted due to blocking budget

This confirms that Karpenter is attempting to consolidate underutilized nodes but is being blocked by your disruption budget.

Monitoring node and pod lifecycle with Amazon CloudWatch

To track when your nodes and pods are created or deleted over the 21-day monitoring period, you can use Amazon CloudWatch Logs and CloudWatch alarms. Amazon EKS automatically sends Kubernetes API server audit logs to CloudWatch Logs when you enable it, which contain detailed information about all resource lifecycle events.

You can create metric filters to extract specific events from these audit logs and trigger alarms when nodes or pods are disrupted. The metric filters use JSON pattern matching to identify creation and deletion events for your specific resources. When these events occur, CloudWatch can send notifications through Amazon Simple Notification Service (Amazon SNS), allowing you to track exactly when disruptions happen.

See example AWS CLI command to create metric filters and alarms for specific node and pod tracking:

# Node Metric Filter
aws logs put-metric-filter \
  --log-group-name /aws/eks/EKS_CLUSTER_NAME/cluster \
  --filter-name NodeLifecycleEvents \
  --filter-pattern '{ (($.verb = "create") || ($.verb = "delete")) && ($.objectRef.resource = "nodes") && ($.objectRef.name = "i-abcde12345abc") && (($.responseStatus.code = 200) || ($.responseStatus.code = 201)) }' \
  --metric-transformations \
      metricName=NodeLifecycleEvents,metricNamespace=EKS/DisruptionTest,metricValue=1,defaultValue=0

# Pod Metric Filter
aws logs put-metric-filter \
  --log-group-name /aws/eks/EKS_CLUSTER_NAME/cluster \
  --filter-name PodLifecycleEvents \
  --filter-pattern '{ (($.verb = "create") || ($.verb = "delete")) && ($.objectRef.resource = "pods") && ($.objectRef.name = "POD_NAME*") && (($.responseStatus.code = 200) || ($.responseStatus.code = 201)) }' \
  --metric-transformations \
      metricName=PodLifecycleEvents,metricNamespace=EKS/DisruptionTest,metricValue=1,defaultValue=0

# Node alarm
aws cloudwatch put-metric-alarm \
  --alarm-name NodeDeletionAlert \
  --alarm-description "Alert when node i-abcde12345abc is deleted" \
  --metric-name NodeLifecycleEvents \
  --namespace EKS/DisruptionTest \
  --statistic Sum \
  --period 60 \
  --evaluation-periods 1 \
  --threshold 1 \
  --comparison-operator GreaterThanOrEqualToThreshold \
  --treat-missing-data notBreaching

# Pod alarm
aws cloudwatch put-metric-alarm \
  --alarm-name PodDeletionAlert \
  --alarm-description "Alert when POD_NAME pods are created or deleted" \
  --metric-name PodLifecycleEvents \
  --namespace EKS/DisruptionTest \
  --statistic Sum \
  --period 60 \
  --evaluation-periods 1 \
  --threshold 1 \
  --comparison-operator GreaterThanOrEqualToThreshold \
  --treat-missing-data notBreaching

Note: Replace POD_NAME, EKS_CLUSTER_NAME and i-abcde12345abc with your own values.

The following image shows a CloudWatch Metric configured to monitor Node Creation or Node Deletion Events in the Amazon EKS cluster. The metric is also configured with an CloudWatch Alarm to trigger a notification whenever these events occur.

For detailed information about setting up CloudWatch Logs metric filters and alarms, see Creating CloudWatch alarms for CloudWatch Logs metric filters in the Amazon CloudWatch Logs User Guide.

Trade-offs to consider

While this approach maximizes node stability, it comes with trade-offs:

Cost efficiency: Blocking consolidation means you might run underutilized nodes longer than necessary. In the test environment, a single node (c6a.xlarge) running at 4.2% CPU utilization costs approximately $133 per month. In production, you must balance stability requirements against compute costs.

Mandatory updates: EKS Auto Mode enforces a 21-day maximum node lifetime to ensure up-to-date software and APIs. Your disruption budget cannot prevent these mandatory updates, so you must plan your application upgrades within this window.

Pod Disruption Budgets: While NodePool disruption budgets control node-level disruptions, you can also use Pod Disruption Budgets to protect individual applications from voluntary evictions during cluster operations like node drains or upgrades. PDBs work at the application layer and complement NodePool disruption budgets by ensuring a minimum number of pods remain available during voluntary disruptions.

Here's an example PDB configuration:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  labels:
    app: non-disruptible
  name: non-disruptible-pdb
spec:
  maxUnavailable: 0  
  selector:
    matchLabels:
      app: non-disruptible

For scenarios to set PDB with maxUnavailable=0 refer to Think about how your application reacts to disruptions

Below are sample workloads running in my test cluster for over 13 days, demonstrating that both approaches successfully prevent disruptions when combined with NodePool disruption budgets:

kubectl get deployment,nodes,pods  
NAME                                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/non-disruptible-app-no-pdb     3/3     3            3           13d
deployment.apps/non-disruptible-app-with-pdb   3/3     3            3           13d

NAME                       STATUS   ROLES    AGE     VERSION
node/i-0b256039e8bfcc04e   Ready    <none>   13d     v1.34.3-eks-3c60543

NAME                                                READY   STATUS    RESTARTS   AGE
pod/non-disruptible-app-no-pdb-667748f9f8-dd4jz     1/1     Running   0          13d
pod/non-disruptible-app-no-pdb-667748f9f8-kcp4b     1/1     Running   0          13d
pod/non-disruptible-app-no-pdb-667748f9f8-zl42h     1/1     Running   0          13d
pod/non-disruptible-app-with-pdb-658bbd7bc6-5s5m6   1/1     Running   0          13d
pod/non-disruptible-app-with-pdb-658bbd7bc6-8nmvr   1/1     Running   0          13d
pod/non-disruptible-app-with-pdb-658bbd7bc6-b9stt   1/1     Running   0          13d

Note: For this test scenario, the deployments non-disruptible-app-with-pdb and non-disruptible-app-no-pdb are scheduled unto the same node which is has NDB configured.

Validating Resource Lifespan and Termination Events

After implementing the Pod Disruption Budgets (PDB) and Node disruption prevention mechanisms, it's important to verify their effectiveness by analyzing how long your resources actually lasted before termination. CloudWatch Logs Insights provides detailed audit logs from the Kubernetes API server that track the complete lifecycle of pods and nodes. You can use the following CloudWatch Logs Insights query to calculate how long your nodes remained running:

fields @timestamp, 
       objectRef.name as ResourceName, 
       verb, 
       coalesce(requestObject.status.nodeInfo.osImage, responseObject.status.nodeInfo.osImage) as OSType
| filter @logStream like /kube-apiserver-audit/ 
| filter objectRef.name = "i-0b256039e8bfcc04e" 
| filter objectRef.resource = "nodes"
| filter verb in ["delete", "create"]
| filter responseStatus.code in [200, 201]
| sort @timestamp desc

Note: Replace the Node name i-0b256039e8bfcc04e with your own Node name. The following image shows aggregated view showing node creation and deletion.

The node i-0b256039e8bfcc04e remained operational for approximately 20 hours between creation and deletion events. This extended runtime demonstrates that:

• Node disruption budget policies are being respected - Pods were not abruptly terminated
• Graceful shutdown procedures executed - The node deletion followed proper drain protocols
• EKS Auto Mode managed the lifecycle - Automatic scaling and maintenance occurred without service disruption

Conclusion

In this post, I showed you how to configure NodePool disruption budgets in Amazon EKS Auto Mode to maximize node and pod lifetimes while respecting AWS's mandatory 21-day update window. By setting a disruption budget of 0% and configuring appropriate expiration times, you can give your team the time needed to test and deploy application updates on your own schedule.

This approach works for both Amazon EKS Auto Mode and self-managed Karpenter in standard EKS clusters, as they share the same underlying NodePool configuration model. The key is understanding the different disruption types and using disruption budgets to control when and how your nodes can be replaced.

To learn more about managing disruptions in Amazon EKS, see the Amazon EKS Auto Mode documentation and the Karpenter disruption concepts.