How do I troubleshoot Amazon ECS tasks that take a long time to stop when the container instance is set to DRAINING?

Short description

When you set an ECS instance to DRAINING, Amazon ECS prevents new tasks from being scheduled for placement on the container instance. Amazon ECS also stops tasks on the container instance that are in the RUNNING state.

Issues with configuration parameters or tasks can keep tasks in the RUNNING state or delay their transition to the STOPPED state.

To troubleshoot these issues, complete the following tasks:

• Update your DeploymentConfiguration parameters
• Update the deregistration delay value
• Update the ECS_CONTAINER_STOP_TIMEOUT value
• Look for other task-related issues

Resolution

To troubleshoot Amazon ECS tasks that take a long time to stop, complete the following tasks.

Update your DeploymentConfiguration parameters

Complete the following steps:

1. Open the Amazon ECS console.
2. In the navigation pane, choose Clusters. Then, choose the cluster where your container instance is draining.
3. Choose the Infrastructure tab.
4. Under Container instances, filter by Status for DRAINING.
5. Choose your container instance, and then find out the service for the tasks that are draining or taking a long time to drain.
6. Choose the Services tab, select the service, and then choose Deployments.
7. Check the values for minimumHealthyPercent and maximumPercent.
   Note: Service tasks on the container instance that are in the RUNNING state are stopped and replaced according to the service's deployment configuration parameters. For more information, see Draining Amazon ECS container instances.

Update the deregistration delay value

Important: The following steps apply only to services that use the Application Load Balancer or Network Load Balancer. If your service uses the Classic Load Balancer, then check the connection draining values.

Complete the following steps:

1. Open the Amazon ECS console.
2. In the navigation pane, choose Clusters, and then choose the cluster where your container instance is draining.
3. Choose the Services tab, and then select the service with the stack that's stuck in RUNNING.
4. Choose Target Group Name.
5. On the Details tab, scroll down, and then select the Deregistration delay check box.

Update the ECS_CONTAINER_STOP_TIMEOUT value

Complete the following steps:

1. Use SSH to connect to your container instance.

2. To find the ECS_CONTAINER_STOP_TIMEOUT value, run the following command:

   docker inspect ecs-agent --format '{{json .Config.Env}}'

3. If there's a value for ECS_CONTAINER_STOP_TIMEOUT, then increase the duration value.
   Note: ECS_CONTAINER_STOP_TIMEOUT is an ECS container agent parameter that defines the amount of time that Amazon ECS waits before ECS ends a container. The time duration starts counting when a task is stopped. If you don't see the ECS_CONTAINER_STOP_TIMEOUT parameter in the output after you run the command, then Amazon ECS is using the default value of 30s.

Look for other task-related issues

To look for other task-related issues, use SSH to connect to your container instance. Then, complete the following tasks:

• Verify that the Docker daemon and Amazon ECS container agent are running for either your Amazon Linux 1 AMIs or Amazon Linux 2 AMIs.
• Check the application logs based on the log driver set by logConfiguration. For example, if your tasks use the awslogs log driver, then check your Amazon CloudWatch Logs for issues.