AWS re:Invent 2024 - Deep dive into Amazon ECS resilience and availability

At AWS re:Invent 2024, Maish Saidel-Keesing (Senior Developer Advocate) and Malcolm Featonby (Sr Principal SDE) presented a deep dive into the resilience and availability strategies of Amazon Elastic Container Service (Amazon ECS). Their session, titled "Deep dive into Amazon ECS resilience and availability," provided valuable insights into how AWS designs and operates one of its core container services to ensure high availability and fault tolerance. This blog post summarizes the key points from their presentation, offering a look into the architectural decisions and operational practices that keep Amazon ECS running smoothly at scale.

Understanding Amazon ECS

To kick off the session, Maish Saidel-Keesing provided a comprehensive introduction to Amazon Elastic Container Service (Amazon ECS). Think of Amazon ECS as a smart manager for your containerized applications. Here's what you need to know:

1. What is Amazon ECS? It's a fully managed container orchestration service that helps run, stop, and manage containers.
2. Impressive stats: Amazon ECS has been around for 10 years and is available in 34 AWS regions. It handles over 2.4 billion tasks weekly, showing its massive scale and reliability.
3. Popularity: 65% of customers starting with containers on AWS choose Amazon ECS. Of those, 70% opt for AWS Fargate, which manages the underlying infrastructure for you.
4. Importance: Amazon ECS is considered a "foundational service" within AWS. This means it's essential for many other AWS services and is always set up before a new AWS region launches.

ECS has evolved significantly over its 10-year history, adapting to the changing needs of cloud computing. Its global availability means developers can run containers close to their users, reducing latency. The integration with other AWS services allows for complex, multi-service architectures. This combination of maturity, reach, and versatility makes Amazon ECS suitable for a wide range of applications, from simple web apps to complex microservices.

Building for Availability: The Three Pillars

Malcolm Featonby then delved into how Amazon ECS is built to ensure high availability. He explained that the team focuses on three main areas, which we can think of as pillars supporting the service's reliability:

1. Infrastructure: This pillar is about the physical and virtual resources that support Amazon ECS.
   • Amazon ECS is deployed across multiple Availability Zones (AZs) in each region.
   • Each AWS region operates independently, creating isolation between geographical areas.
   • The service is pre-scaled to 150% capacity across AZs, providing a buffer for unexpected load or AZ failures.
2. Software: This pillar focuses on how the Amazon ECS software is designed and updated.
   • Amazon ECS uses a cellular architecture, dividing the control plane into isolated units called "cells".
   • Changes are rolled out gradually, starting with a single cell in one AZ.
   • The team employs careful monitoring and automated rollback mechanisms during deployments.
3. Scale: This pillar ensures Amazon ECS can handle massive growth and varying workloads.
   • The cellular architecture allows for easier scaling by adding more cells as needed.
   • Amazon ECS is designed to handle sudden spikes in demand without degradation of service.
   • The team regularly conducts "scale to break" tests on individual cells to understand system limits.

Malcolm emphasized that these pillars work together to create a robust system. For example, the infrastructure design allows for software updates to be rolled out safely, while the scaling mechanisms ensure the infrastructure can handle growing demands.

He also noted that many of these principles are available to Amazon ECS users through features like multi-AZ task placement, rolling updates, and auto-scaling. This allows customers to build their own highly available applications on top of Amazon ECS.

Smart Architecture Choices

Malcolm Featonby dove deeper into the specific architectural decisions that make Amazon ECS resilient. These choices go beyond the basic pillars and show how AWS puts availability principles into practice:

1. Blast Radius Containment: A sharding approach distributes customer workloads across different partitions. This ensures that problems in one shard don't affect others, protecting most customers from any single issue.
2. Control Plane vs. Data Plane Separation: The architecture separates the control plane (task management) from the data plane (task execution). This allows independent scaling and ensures tasks keep running even if the control plane has issues.
3. Eventual Consistency Model: The system uses eventual consistency, allowing it to operate even when some parts are temporarily unavailable. This trades immediate consistency for improved availability and performance.
4. Automated Recovery Mechanisms: Self-healing capabilities, like automatically restarting failed tasks, are built-in. This minimizes downtime and reduces the need for manual intervention.
5. Dependency Isolation: Dependencies on other AWS services are minimized where possible. When dependencies exist, they're carefully managed to prevent cascading failures.

Malcolm emphasized that these choices reflect lessons from years of large-scale operations, encouraging attendees to consider similar principles in their own designs, regardless of scale.

Careful Deployment Strategies

Malcolm Featonby then explained how Amazon ECS implements careful deployment strategies to ensure high availability during updates:

1. Gradual Rollouts: Changes are rolled out incrementally, starting with a single cell in one Availability Zone. This approach limits the potential impact of any issues that might arise during deployment.
2. Bake Time: After deploying to a subset of the infrastructure, ECS uses a bake time to monitor for any problems. This period allows for thorough testing and observation before expanding the deployment.
3. Automated Rollbacks: If issues are detected during deployment, ECS can automatically roll back to the previous known-good state. This quick response helps minimize potential downtime.
4. Deployment Isolation: The cellular architecture allows for deployments to be isolated to specific cells, further reducing the blast radius of any potential issues.
5. Version Stability: A new feature ensures that when scaling or rolling back, ECS always uses a consistent, known-good version of task definitions and configurations.

Malcolm emphasized that these strategies work together to allow for continuous improvement of the service while maintaining high availability. He also noted that many of these practices are available to ECS users through features like rolling updates and blue/green deployments.

Automation for Resilience

Malcolm Featonby highlighted how Amazon ECS leverages automation to enhance its resilience:

1. Automated Weigh-Away: ECS continuously monitors the health of each Availability Zone. If issues are detected in an AZ, the system automatically redirects traffic and workloads away from it, ensuring continued service availability.
2. AZ Rebalance: A new feature automatically rebalances tasks across healthy AZs when an imbalance is detected. This maintains optimal distribution of workloads without manual intervention.
3. Local Container Restart: When a container fails, Amazon ECS attempts to restart it locally before initiating a full reprovisioning. This faster recovery method helps maintain service continuity.
4. Non-Blocking I/O for Logging: Amazon ECS supports non-blocking I/O for container logs. This prevents application slowdowns or failures due to logging issues, enhancing overall system resilience.
5. Predictive Scaling: Using machine learning, Amazon ECS can predict upcoming load patterns and preemptively scale resources, ensuring smooth handling of traffic spikes.

Malcolm emphasized that these automated features work together to create a self-healing system that can quickly respond to and recover from various types of failures. He encouraged users to leverage these capabilities in their own Amazon ECS deployments to improve application resilience.

Always Learning and Improving

Maish Saidel-Keesing then explained how Amazon ECS continually evolves through a process of learning from experiences:

• Chaos Experiments: The team regularly conducts "game days" where they intentionally introduce failures to test system resilience. These controlled experiments help identify weak points and improve recovery processes.
• Correction of Errors (COE) Process: After any significant incident, the team follows a structured COE process. This involves a detailed analysis of what happened, why it happened, and how to prevent similar issues in the future.
• Operational Metrics: ECS continuously collects and analyzes operational metrics. These data-driven insights guide improvements in system design and operational procedures.
• Customer Feedback Loop: The team actively seeks and incorporates customer feedback. This helps them understand real-world use cases and challenges, informing future feature development.
• Post-Incident Learning: Maish shared two specific examples of how past incidents led to significant improvements:
  • A 2024 Kinesis outage resulted in the development of non-blocking I/O options for ECS tasks.
  • A 2022 Fargate issue led to improvements in the cellular architecture and the introduction of automated weigh-away features.

Maish emphasized that this culture of continuous learning and improvement is key to maintaining and enhancing the reliability of ECS. He encouraged attendees to adopt similar practices in their own organizations, regardless of scale.

Wrapping Up

Maish and Malcolm concluded by emphasizing the key aspects of Amazon ECS's resilience and availability. They highlighted how ECS leverages AWS's global infrastructure and operational experience to maintain high availability. The speakers stressed the importance of architectural choices like cellular design and blast radius containment, as well as the critical role of automation in maintaining system health.

They underscored Amazon's culture of learning from failures, sharing examples of how past incidents led to significant improvements in Amazon ECS. The speakers encouraged attendees to apply these principles in their own systems, reminding them that many of the discussed resilience features are available to ECS users. Finally, they shared supplementary resources for continued learning. For those interested in diving deeper, the full session recording is available on the AWS YouTube channel.