EC2 Instance Terminated Itself, Termination Protection was Enabled

Question

I have a scary situation that is currently unfolding. Our production server, which has been running for nearly three years, suddenly terminated itself. We have no idea why or more importantly how since the instance had termination protection enabled. As soon as I learned the server was down I checked the details in AWS and saw that somehow termination protection had been changed to disabled.

How is any of this possible? Why would an instance terminate itself? Isn't the point of termination protection to make sure this doesn't happen?

I've now got a production environment completely offline and I have no idea why. I'm fortunate that my senior developers were able to jump right in, but they are not AWS experts. So who knows how long it will take to get everything back online.

I'd just really like to know how this could happen. Especially with the amount of money I pay each month. I'm just dumbfounded.

Answer

I would take a look at the CloudTrail logs ASAP to determine which principal disabled termination protection. If you don't have additional security tooling setup, can you view CloudTrail events in the Console: https://docs.aws.amazon.com/awscloudtrail/latest/userguide/view-cloudtrail-events-console.html

Answer

**Termination protection** in AWS EC2 instances helps prevent accidental deletions but doesn't cover all scenarios. It doesn't shield against scheduled maintenance, Auto Scaling actions, or terminations of Spot Instances. In your case, if your instance was undergoing maintenance or part of an Auto Scaling group, termination could still occur despite protection being enabled.

So I guess you instance were under scheduled maintenance.
![Scheduled Maintenance](/media/postImages/original/IMAIqByp1GS_KG7uv67rQZnQ)

Resources:
* [Enable termination protection - Limitation](https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/terminate-instances-considerations.html#Using_ChangingDisableAPITermination)

EC2 Instance Terminated Itself, Termination Protection was Enabled

関連するコンテンツ