How do I configure my capacity provider to retain Amazon EC2 instances in an Amazon EC2 Auto Scaling group that has running Amazon ECS tasks?

Short description

If you use a capacity provider, then you must configure your Amazon ECS cluster settings to not disrupt running tasks when it scales down. Implement scale-in instance protection in the EC2 Auto Scaling group and managed scaling protection in the Amazon ECS capacity provider settings. This configuration makes sure Amazon ECS retains instances with running tasks.

Or, configure Amazon ECS to gracefully drain tasks when you stop instances, and protect specific instances from being prematurely stopped.

Your configuration uses the following components:

• Capacity providers: Capacity providers use Auto Scaling groups to dynamically provision EC2 instances.
• Instance draining: When an EC2 instance in an Auto Scaling group is marked for termination, it enters a DRAINING status and no longer accepts new tasks. Running tasks must either finish or move to another instance.
• Cluster auto scaling: Cluster auto scaling uses capacity providers to automatically scale Auto Scaling groups based on task requirements.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Make sure that Amazon ECS or the Auto Scaling group doesn't prematurely terminate an instance during scale in

To use managed termination protection in cluster auto scaling, you must activate instance scale-in protection on the Auto Scaling group. This action makes sure that the Auto Scaling group retains instances with running tasks.

To activate scale-in protection, run the following update-auto-scaling-group AWS CLI command:

aws autoscaling update-auto-scaling-group \
  --auto-scaling-group-name my-asg \
  --new-instances-protected-from-scale-in

Note: Replace my-asg with your Auto Scaling group name.

Then, run the following put-cluster-capacity-providers command to activate managed termination protection to make sure that Amazon ECS retains instances with running tasks:

aws ecs put-cluster-capacity-providers \
  --cluster my-cluster \
  --capacity-providers my-capacity-provider \
  --enable-managed-scaling \
  --managed-termination-protection ENABLED

Note: Replace my-cluster with your cluster name and my-capacity-provider with your capacity provider name.

Configure Amazon ECS to gracefully drain tasks when you remove an instance

It's a best practice to gracefully drain tasks when you remove them from an instance. Use managed instance draining to allow Amazon ECS to reschedule your tasks on other instances before termination.

To activate managed instance draining, run the following update-capacity-provider command:

aws ecs update-capacity-provider \
--name my-cp \
--auto-scaling-group-provider '{
  "managedDraining": "ENABLED"
}

Note: Replace my-cp with your capacity provider name.

Amazon ECS moves tasks on the instance to other instances as capacity allows. However, Amazon ECS allows maximum 48 hours to drain instances. If there are still tasks on the instance after this period, then Amazon ECS terminates the instance and stops the remaining tasks.

Retain a specific critical task

Use task protection to make sure that Amazon ECS scaling or deployment actions don't stop critical tasks.

Important: Even when you activate task protection, Amazon ECS can still terminate the underlying instance. Task protection only protects tasks from being stopped.

To activate task protection, run the following update-task-protection command:

aws ecs update-task-protection \
--cluster my-cluster \
--tasks my-task-id \
--protection-enabled

Note: Replace my-cluster with your cluster name and my-task-id with your task ID. By default, protection lasts for 2 hours. To change this duration, use the --expires-in-minutes option.

To check the status of task protection in your cluster, run the following get-task-protection command:

aws ecs get-task-protection \
--cluster my-cluster \
--tasks my-task-id

Note: Replace my-cluster with your cluster name and my-task-id with your task ID.

Troubleshoot further issues

If you activate protection for your Auto Scaling group, capacity provider, and tasks, and still encounter issues, then take the following actions:

• Check the Amazon ECS agent logs for task draining issues.
• Make sure that the Amazon ECS agent is running and correctly configured on your instances.
• Verify that your AWS Identity and Access Management (IAM) task roles have the permissions that the containers in your tasks must assume.

Related information

Protect your Amazon ECS tasks from being terminated by scale-in events

GetTaskProtection

UpdateTaskProtection

Amazon Elastic Container Service (Amazon ECS) task protection examples on the GitHub website