- Newest
- Most votes
- Most comments
There are only 2 situations[1] where the ASG can change its own max capacity, neither of which should apply to your situation. I suggest looking in cloudtrail for EventName=UpdateAutoScalingGroup to see if maybe there was a user or some sort of automated script which increased the max?
Also just to confirm: Did the Max actually get changed from 5 to 31, or were there 31 total instances, but the max was still set to 5? Its possible if there were frequent healthcheck replacements happening from the instance CPU being maxed out, that the ASG could have had more instances than the max. The Max is a boundary for the desired capacity, not the total instances. While an instance is being replaced due to healthcheck failures, it isn't going to contribute towards the groups capacity calculations, and therefore the total instance count can be over max temporarily
[1]
- Scheduled scaling: If a scheduled action is created, it can be set to change the min, max, and/or desired capacity of the ASG. You would have had to explicitly created this, so I doubt its applicable here
- PredictiveScaling when the MaxCapacityBehavior is set to IncreaseMaxCapacity. However, Predictive Scaling only changes the groups capacity based on the past 1-14 days of usage. It does not react to any real time changes to usage in the group like you described
Relevant content
- asked 2 years ago
- asked 5 months ago
- AWS OFFICIALUpdated 2 months ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 4 months ago
Thanks for the answer Shahad. I'm unable to verify if MaxCapacity indeed changed from 5 to 31 since I never enabled monitoring, but what you described - healthcheck replacements happening due to CPU being maxed out - is most likely what happened.
If you don't see the max as 31 currently; and you don't see any UpdateAutoScalingGroup calls in CloudTrail lowering it back down; then its a pretty safe bet it didn't change. AutoScaling metrics are free, so I'd recommend enabling them now to make future troubleshooting simpler: https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-metrics.html