Is it expected behavior that an additional ECS deployment occurs and causes downtime?

Question

Hello, we are using ECS on EC2.

After an ECS deployment completes, another deployment automatically occurs about 15–20 minutes later, and this process causes downtime.
We would like to know whether this is expected behavior or a bug.
If it is expected behavior, we would also like to know how to prevent this issue.

Here is our setup:
* The ECS cluster uses a capacity provider that is connected to an Auto Scaling Group (ASG).
* The ASG is configured to allow 2–4 EC2 instances.
* Our ECS service runs 2 tasks and is configured to use a 100–200% rolling update deployment.
* Each ECS task is configured so that one task fully utilizes one EC2 instance (CPU and memory-sized appropriately).
* Therefore, during deployment, the number of tasks scales up to 4, and the number of EC2 instances also scales up to 4.

About 15–20 minutes after the deployment completes, it appears that the capacity provider scales the number of instances back down to 2.
During this process, we expect instances without running tasks to be terminated first.
However, instances with running tasks are terminated without proper draining, which causes service downtime.
The downtime continues until the tasks are rescheduled on the remaining instances.

If we manually enable scale-in protection on instances running tasks and then reduce the ASG desired capacity to 2, the ASG correctly terminates only the empty instances and no downtime occurs. (We disable scale-in protection afterward.)

Below is our environment configuration:
* Instance type: t3a.small
* Task definition: CPU 750 / Memory 1800
* AZ Rebalancing: Enabled
* Capacity provider managed scaling: Enabled
* Capacity provider managed draining: Disabled (Could this be the cause?)

Please let me know if you need any additional information. Thank you.

re:Post Agent · Accepted Answer

What you're experiencing is not expected behavior for ECS deployments. The downtime you're seeing approximately 15-20 minutes after deployment is related to how your capacity provider and Auto Scaling Group are handling scale-in operations.

The issue appears to be that when your capacity provider scales the number of EC2 instances back down from 4 to 2 after deployment, it's not properly respecting task placement or ensuring instances with running tasks are drained before termination. This is causing your service downtime.

The key factor here is likely that you have "Capacity provider managed draining" disabled. This feature is designed specifically to prevent the scenario you're experiencing. When enabled, it ensures that:

1. Instances with running tasks are properly drained before termination
2. Tasks are migrated to other instances before the host instance is terminated
3. Scale-in operations target empty instances first

I recommend enabling "Capacity provider managed draining" as this should resolve your issue. This feature automatically manages instance scale-in protection for you, similar to what you're doing manually.

Additionally, you might want to check:

1. Your ASG cooldown periods - ensure they're not too short, which could cause rapid scaling decisions
2. The capacity provider's target tracking metrics - ensure they're appropriate for your workload
3. Consider adjusting your minimum healthy percent and maximum percent settings if you need to further optimize the deployment process

The behavior you're seeing when manually enabling scale-in protection (no downtime) confirms that the issue is related to instance termination during scale-in rather than the ECS deployment process itself.  
**Sources**  
[The rollout update of new ECS service version is slow | AWS re:Post](https://repost.aws/questions/QUtuJWfRoeTuK6AKIIFPOSjA/the-rollout-update-of-new-ecs-service-version-is-slow)  
[The rollout update of new service version is slow | AWS re:Post](https://repost.aws/questions/QUtuJWfRoeTuK6AKIIFPOSjA/the-rollout-update-of-new-service-version-is-slow)  
[ECS Task Replacement | AWS re:Post](https://repost.aws/questions/QU49LPh3RWRgaUw0gLHMfN0g/ecs-task-replacement)

Is it expected behavior that an additional ECS deployment occurs and causes downtime?

Add your answer

Relevant content