Why did the running task count change in my Amazon ECS service?

Short description

The running task count for your Amazon ECS service might increase or decrease for the following reasons:

• Your Amazon ECS tasks are failing health checks.
• AWS CloudFormation or AWS Cloud Development Kit (AWS CDK) updated the desired count in the ECS service.
• You manually updated the desired count for your ECS service.
• The Amazon ECS Service Auto Scaling updated the number of tasks based on the scaling policy.
• You used the RegisterScalableTarget API to update the minimum or maximum count in Application Auto Scaling.
• The deployment configuration of your ECS service modified the running task count during deployment.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshoot AWS CLI errors. Also, make sure that you're using the most recent AWS CLI version.

Based on your use case, complete the following tasks to determine why your running tasks count changed.

Your ECS tasks are failing health checks

Complete the following tasks:

• If your tasks use the Amazon EC2 launch type, then see How can I get my Amazon ECS tasks that use the Amazon EC2 launch type to pass the Application Load Balancer health check?
• If your tasks use the AWS Fargate launch type, then see How do I troubleshoot health check failures for Amazon ECS tasks on Fargate?

CloudFormation or AWS CDK updated the desired count in your ECS service

If you created your ECS service with CloudFormation or CDK and didn't specify the DesiredCount, then DesiredCount is set to the default of 1. When you use CloudFormation or CDK to make updates without specifying the DesiredCount field, then the current deployment is updated. The existing desired count is used to update the count in the current deployment.

To troubleshoot this issue, review AWS CloudTrail events to check if the UpdateService API call was made by CloudFormation. Or, to verify this same issue, check that the userAgent field in the Event record section specifies the following:

"userAgent": "cloudformation.amazonaws.com"

To resolve this issue, remove the desiredCount parameter from the CloudFormation or CDK code during the update.

You manually updated the desired count for your ECS service

If you use the UpdateService API, ECS console, or AWS CLI to manually update the ECS Service, then update the service again. Update the service to change the desired count to an appropriate value. For more information, see UpdateService and update-service.

To troubleshoot this issue, review CloudTrail events to check whether the UpdateService API call was made from the Amazon ECS console or AWS CLI. View this information in the userAgent field in the CloudTrail record.

Example:

"userAgent": "console.amazonaws.com"

The Amazon ECS Service Auto Scaling updated the number of tasks based on the scaling policy

The Amazon ECS Service Auto Scaling might have increased or decreased the desired count based on the scaling policy. To review the service and check for recent scaling activity, complete the following steps:

1. Open the Amazon ECS console.
2. In the navigation pane, choose Clusters, and then choose the cluster that you want to check.
3. Choose the Services tab, and then choose the service that you want to check.
4. Choose the Events tab.

If a scaling event occurred, then you see a message in the Message column similar to the following:

"Successfully set desired count to 1. Change successfully fulfilled by ecs. Cause: monitor alarm TargetTracking-service/service-autoscaling/sample-webapp-AlarmLow-fcd80aef-5161-4890-aeb4-35dde11ff42c in state ALARM triggered policy TargetTrackingPolicy."

If the tasks are stopped because of a scaling activity, then review the target tracking or step scaling policies. Then, update the desired count based on your application's requirements.

To view the Application Auto Scaling configuration, complete the following steps:

1. Open the Amazon ECS console.
2. In the navigation pane, choose Clusters, and then choose the cluster that you want to check.
3. Choose the Services tab, and then choose the service that you want to check.
4. Choose the Auto Scaling tab.

You used the RegisterScalableTarget API to update the minimum or maximum count in Application Auto Scaling

If you use the RegisterScalableTarget API to update the scalable target, then check the MinCapacity property value specified in the API. If you specified a new minimum value, then Application Auto Scaling uses this value for what tasks can be scaled down to.

Note: Application Auto Scaling doesn't scale down the desired count to less than the specified minimum value.

The deployment configuration of your ECS service modified the running task count during deployment

To check if the UpdateService API was recently called, review CloudTrail events. To view the deployment configuration of your ECS service, check the deploymentConfigration request parameter in the UpdateService API.

-or-

To review the deployment configuration of your ECS service in the Amazon ECS console, complete the following steps:

1. Open the Amazon ECS console.
2. In the navigation pane, choose Clusters.
3. Choose the cluster that you want to check.
4. Choose the Services tab.
5. Choose the service that you want to check.
6. Choose the Deployments tab.

The deployment configuration parameters control how many tasks are run during a deployment and the sequence of stopping and starting tasks. If the deployment configuration is incorrectly set, then the number of tasks that run during the deployment is reduced. Therefore, it's a best practice to configure the value of the minimumHealthyPercent parameter to higher than 0%.

The minimumHealthyPercent parameter represents the lower limit on either of the following as a percentage of the desired number of tasks for the service:

• Number of tasks that run for a service during a deployment
• Number of tasks that run when a container instance is draining

The value of this parameter is rounded up. For example, if minimumHealthyPercent is 50% and the desired number of tasks is 4, then the scheduler can stop two existing tasks. Then, the scheduler can start two new tasks.

If minimumHealthyPercent is 75% and the desired number of tasks is 2, then the scheduler can't stop any tasks because the resulting value is 2.

The maximumPercent parameter represents the upper limit on either of the following as a percentage of the desired number of tasks for a service:

• Number of tasks that run for a service during a deployment
• Number of tasks that run when a container instance is draining

The value of this parameter is rounded down. For example, if maximumPercent is 200% and the desired number of tasks is 4, then the scheduler can start 4 new tasks. Then, the scheduler can stop 4 existing tasks. If maximumPercent is 125% and the desired number of tasks is 3, then the scheduler can't start any tasks because the resulting value is 3.

For more information, see Deploy Amazon ECS services by replacing tasks.

Related information

Automatically scale your Amazon ECS service

How do I resolve the "no container instance met all of its requirements" error in Amazon ECS?