Optimizing your Amazon EKS operations with Unified Operations

Introduction

The flexibility, scalability, and portability of containerized workloads have made Amazon EKS a widely used service for businesses across industries. Amazon EKS powers everything from microservice architectures to machine learning workloads, as seen with a 33% year-over-year increase in Amazon EKS adoption. As these workloads scale operations, teams often need to react to problems rather than focus on innovation and growth. The complexity of Kubernetes operations requires a more systematic approach.

To help customers address these challenges, AWS Support has built Unified Operations to provide customers with a framework specifically designed for operational excellence at scale. Unified Operations combines the following features:

• Enhanced observability
• Rapid incident response
• Regular architecture reviews
• Continuous testing
• Specialized event support

These features create a continuous improvement cycle that enhances resiliency and allows teams to focus on innovation and business value.

The Amazon EKS operational challenge

A common example scenario

The following example is a common scenario from AWS Support customers. A development team uses containers to successfully modernize their application and deploy to Amazon EKS. The initial launch goes smoothly, but within weeks, operational issues begin to surface:

• Performance degrades during peak traffic and cascades throughout the application
• Mysterious "stuck pod" incidents require manual intervention
• Etcd latency spikes cause cluster-wide instability
• Scaling issues prevent proper handling of traffic surges
• Resources are unexpectedly exhausted despite apparent capacity

The operations team, who initially plans for upcoming Amazon EKS upgrades and innovation initiatives, instead must constantly address issues. Incident response becomes the team’s primary focus, and they have little time for proactive improvements or strategic planning.

The root cause

These challenges often don’t come from technical limitations of Amazon EKS itself, but from gaps in operational practices:

• Observability gaps: Traditional monitoring approaches fail to provide the multilayered visibility that Kubernetes environments need.
• Reactive posture: Without proper testing and architecture reviews, teams can only respond to issues after they affect users.
• Knowledge gaps: Amazon EKS operational excellence requires specialized expertise that many teams are still developing.
• Lack of systematic approach: Unplanned operations without a framework lead to inconsistent practices and recurring issues.

Solution overview: The Unified Operations framework

What is Unified Operations?

AWS Support has worked and supported customers through their modernization journeys for many years. By working with customers, AWS Support built Unified Operations as a comprehensive support and operational excellence program. Unified Operations extends beyond traditional support models and includes Domain Specialist Engineers (DSEs) who develop deep knowledge of your specific environment and business requirements.

Unified Operations uses five key pillars to help customers address these challenges and transform operations from reactive troubleshooting to proactive optimization. This framework uses the expertise of DSEs to help customers apply best practices to their specific environments. The Unified Operations solution combines enhanced observability, rapid incident response, regular architecture reviews, continuous improvement through testing, and specialized event support to create a holistic support framework.

The following image shows the operational excellence improvement lifecycle.

Figure 1: Unified Operations Operational Excellence Framework

Pillar 1: Comprehensive observability

The foundation of operational excellence in Amazon EKS environments is comprehensive observability that goes beyond traditional monitoring approaches. Amazon EKS DSEs help define custom observability strategies tailored to your environment.

These strategies include guidance on the implementation of a multilayered monitoring and observability approach through metric collection, log aggregation, distributed tracing, and alerts.

The following is an example strategy:

Control plane observability

With Amazon EKS enhanced Kubernetes control plane observability, you can automatically display curated dashboards. These dashboards display key control plane metrics within the Amazon EKS console.

Node fleet and resource management

Because of the dynamic nature of Amazon EKS node fleets, you can implement predictive node health monitoring to prevent NotReady states. You can also configure automated responses to common node issues and optimize node autoscaling parameters. It’s a best practice to include pod density monitoring per instance type, maximum pod capacity tracking per node, and cluster-wide resource use tracking.

Enhanced CloudWatch Container Insights integration

Amazon CloudWatch Container Insights with enhanced observability provides granular health, performance, and status metrics up to the container level and control plane metrics. With accelerated compute observability capabilities, you can gain insights into the efficiency of your deep learning and inference algorithms.

Application performance monitoring

The CloudWatch Observability EKS add-on pairs CloudWatch Application Signals with Container Insights and provides application performance telemetry. You can then have a comprehensive view of the application and identify application effects early and quickly.

With observability tuned to your requirements, the next step is to focus on incident response.

Pillar 2: Rapid incident response

Key aspects of a good incident response flow include:

• A rapid response that’s enriched with contextual information about the specific setup
• Predefined runbooks that help with resolution
• A process to capture post-incident learning

With Unified Operations, AWS uses the AWS Incident Detection and Response capability to implement rapid incident response. This capability aims to provide a 5-minute response from AWS Support for critical business incidents. DSEs help build critical incident response runbooks and guidance during the incident. These runbooks and guidance combine the support of expert engineers with a deep understanding of your setup to assist you during critical incidents.

After the incident is complete, customers work with the DSE to review the critical incident and develop a long-term remediation plan for continuous improvements.

Pillar 3: Regular architecture reviews

The focus of this pillar is on architecture review for performance, resilience, cost optimization, and security. This pillar can also include maintenance with updates and upgrades of the various components for the setup. The incident reports from the previous pillar form one of the key inputs within the reviews completed in this pillar.

Unified Operations delivers an initial review of the current setup and conducts ongoing reviews with DSEs. Unified Operations also uses AI tools that are trained on the deep understanding and large-scale experience of a service within AWS to help with these reviews. Reviews can focus on the following:

• Control plane performance analysis based on actual usage patterns, rather than default settings
• CNI configuration optimization to prevent IP exhaustion
• Cluster Autoscaler tuning to minimize node churn
• Resource allocation strategies to optimize pod density and node use

A cost optimization analysis provides the following improvements:

• Right-sizing recommendations based on actual resource consumption patterns
• Reserved Instance and Savings Plan optimization for predictable workloads
• Spot Instance integration strategies for fault-tolerant applications
• Resource waste identification and elimination through systematic analysis

A security review looks at the following configurations:

• Network policy implementation to control pod-to-pod communication
• Pod security standards enforcement to prevent privilege escalation
• Container image scanning to identify and remediate vulnerabilities
• Runtime security monitoring to detect and respond to threats

The DSE helps you adopt and apply best practices across your AWS environment. After you complete a review, regularly test the suggested changes for operational processes to make sure that the environment is robust and continues to work.

Pillar 4: Continuous testing

AWS GameDays and structured chaos engineering represent critical components of operational excellence in Amazon EKS environments. AWS has numerous best practices and guides that DSEs can help map to your use case.

Some of the common best practice approaches include:

• Simple failure scenarios, such as basic pod failures and scaling tests that use AWS Fault Injection Service. You can do this testing with something as small as a pod failure, as seen in the following example:

AWS Fault Injection Simulator experiment for pod failure
apiVersion: fis.aws/v1
resource:
  type: aws:eks:pod
  parameters:
    clusterIdentifier: my-cluster
    namespace: default
    labelSelector: app=my-application
action:
  type: aws:eks:pod-delete
  parameters:
    maxPods: 1
    percentage: 25

• Complex scenarios that progress to node failures and cluster-wide impact testing
• Availability Zone testing that uses Amazon Application Recovery Controller (ARC) zonal shift to simulate complete Availability Zone failures and verify resilience

It’s important to understand that with the best designs, optimizations, and resiliency reviews, you must complete robust testing and practice to know that they work. When you test regularly, you build the muscle memory for when an incident occurs.

Pillar 5: Event support

The focus of this pillar addresses large-scale change. Large changes and significant business events lead to usage spikes, and operations must be ready for these changes. Within Unified Operations, customers work with DSEs with specialized skills to help customers with the following:

• Enhanced monitoring during critical operations
• Expert availability for major upgrades, migrations, or scaling events
• Proactive risk mitigation

These events contribute to improved observability to prepare for events and continue the Operational Excellence Framework.

Implementing Unified Operations in your organization

The Unified Operations implementation approach follows a phased strategy that any organization can take to achieve operational excellence. Through this approach, organizations can transform their Amazon EKS operations from reactive to proactive through a systematic implementation plan. This plan includes the following phases:

1. Assessment phase: Evaluation of current operational maturity and gap identification across all pillars.
2. Implementation phase: Systematic deployment of observability, automation, and process improvements.
3. Continuous improvement: Regular review cycles, feedback loops, and metric tracking to continuously optimize.

For your organization, you can also combine Amazon EKS Auto Mode, AWS Systems Manager, CloudWatch, and AWS specialist support programs to create comprehensive operations management capabilities.

Conclusion

Through systematic implementation of the Unified Operations framework, organizations can transform their Amazon EKS operations from reactive to proactive optimization. This change can improve high availability and performance, and build organizational capabilities for long-term operational excellence. With five pillars of operational excellence within Unified Operations, organizations can create a holistic approach that drives continuous improvement and allows teams to focus on innovation. For more information about Unified Operations, visit Unified Operations or contact your AWS account team.

About the authors

Yomesh Shah

Yomesh Shah is a Senior Solutions Architect at AWS. He brings 25 years of experience helping enterprises maximize the value of their IT investments through optimization, automation, and process improvement. He currently helps AWS customers use and apply scalable AWS Support solutions. Yomesh also holds a patent for the design of a Managed Services control plane in the cloud (US11856055B2).