AWS ECS EC2-backed auto scaling scale-in is not working

Question

Hi,

I have configured AWS managed cluster auto scaling and service auto scaling based on target tracking policies.

1. For AWS EC2 scaling capacity provider has set target capacity to 100% and if the CapacityProviderReservation is above 100% it will launch another EC2 instance which is working fine for me.

2. For AWS ECS service scaling I'm using ECSServiceAverageMemoryUtilization metric and the threshold of 85% was met, a new task was launched. However, there were no available resources to run the new task, causing the cluster capacity to reach 100%. At that point, a new instance was launched. This process worked perfectly — the scale-out action functioned as expected.

The Service Auto Scaling Group (ASG) settings are:
Minimum number of tasks: 1
Maximum number of tasks: 3
ECS service metric: ECSServiceAverageMemoryUtilization at 85%
Target value:   85%
Scale-out cooldown period: 300 (sec)
Scale-in cooldown period: 120 (sec)

Turn ON scale-in

Now, for the past 2 hours, memory utilization has been consistently below the threshold value, and the associated alarm remains in the "In Alarm" state. However, the task has not been stopped or terminated.

If task stopped and  then target capacity will reduce and it will be 100% in this case EC2 instance will not terminated.
I need full guidance in configuration of this values in order to achieve the Auto scale IN and OUT for AWS ECS  EC2 backed cluster

Shahad_C · Answer

You can check why scale-in didn't happen via the steps here:
https://docs.aws.amazon.com/autoscaling/application/userguide/application-auto-scaling-scaling-activities.html#include-not-scaled-activities-with-the-aws-cli

With a very small service like this, its likely that the metric wasn't low enough.  If AutoScaling calculates that removing 1 task will bring the metric back over the target (85% here) then it will choose not to scale-in, even when the Alarm is going off (since if it did scale-in, it would likely result in an immediate scale-out from the increased load on the remaining tasks)

If you don't want these safety features, then you'll need to configure Step scaling instead - however, this will likely result in a less stable environment.  Alternatively, if you think scale-in is safe at this point and another metric would better represent this, then you can consider changing the target tracking metric and/or adding another policy tracking the new metric (and disabling scale-in on the Memory policy)

AWS ECS EC2-backed auto scaling scale-in is not working

Add your answer

Relevant content