AMI Not found or Deleted or Made private

Question

I was briefed that the application  hosted on one of our EC2 instances was down five days ago and logged into the aws console to check what the issue was. I tried logging via ssh into the instance with the issue. It kept failing so I decided to login to the console.

After logging the console manually restarted the server and also tried connecting using instance connect. I was also greeted with this message below:

Failed to connect to your instance
Error establishing SSH connection to your instance. Try again later.

I then checked the description on the instance and saw the following error sent as an attachment to this message. That then prompted me to take the following steps below.
I checked the AMI's we have and can see that nothing was deleted. I checked the Recycle bin and found nothing. I also checked the cloud trail logs and can see that there was no "DeregisterImage" Event in the logs to show that anyone tried or deleted any ami.

I tried creating a new AMI from a recent snapshot to bring the server back to work and that still didn't work as I was also presented with the same error of not being able to ssh into the server. I also tried with a backup of the volume I had to bring back the server into service and still get the same error message as shown in the attachment.

I was now able to create the backup of the instance and access it via the assigned public ip addresses. since the instance i was trying to restore had an elastic ip address assigned to it before i attached the elastic ip address and found that i could no longer access the web application nor ssh into the same instance. When i remove the elastic ip, i am able to view the application using the auto assigned public ip address and also ssh into the instance.

I go to Route 53 to check the hosted zone and nothing has been changed there at all. I reassign the Elastic ip back to the original instance and try calling the web application and it does nothing. All instances we have on the account in the region with my test suddenly become unavailable whenever i associate the Elastic Ip with them.

Can someone help out with this as i think this is the only thing now hindering me from being able to change/replace the instance giving an issue and keeping our application offline for almost a week now.

Answer

Hello,

Greetings!

Thank you for reaching out. I understand that your application on one of your EC2 instances was down, however you were not able to SSH login to the instance to be able to check the issue. Neither the EC2 connect option worked. You received some error message following which you confirmed that no one had deleted your AMIs. Further, you tried creating a AMI from the backup snapshot and launched a backup instance. However, as you attached the Elastic IP address to the instance, the application and SSH didn't work, while things worked with auto-assigned Public IP address. Please let me know if this is not the correct understanding.

Primarily, please allow me to highlight that I did NOT find any file attachment to this message, due to which I am not able to understand in what context the further steps were taken by you.

Could you check if your original instance is passing 2/2 status checks? In case the instance shows 1/2 status checks, please refer to the following documentation to troubleshoot the issue:

Troubleshoot instances with failed status checks
	https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/TroubleshootingInstances.html

Yet if it is passing both the checks, regarding the SSH login issue, what error message do you get when you try to login, say via Terminal or PuTTY? 
Based on the error message, the corresponding remediation steps can be taken.

Troubleshoot connecting to your instance
	https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/TroubleshootingInstancesConnecting.html

However, to be able to exactly answer your question on why SSH fails on the original or the backup server (after attaching the Elastic IP), we require details that are non-public information (such as your EC2 instance IDs, error details etc) to investigate the issue. Thus, I would suggest to please open a support case with AWS using the following link:

https://console.aws.amazon.com/support/home#/case/create

I hope that the above information is helpful.

Thanks again for reaching out to us, we wish you a wonderful day ahead!

Answer

Thanks for the reply @Raghav_G. The instance was behaving in a manner i cant put into words. when i use the publicly assigned ip addresses, i can ssh and do what ever i wanted but when i used the elastic ip address, nothing worked.

So i thought of getting a new elastic ip address and added it into the configurations on route53 and everything was back to normal.

I can say confidently that it would seem that something was wrong with that elastic ip and simply using a newly assigned one solved my issues.

As regards support from aws, I opened a ticket 4 days ago and just only got a response today for such a mission critical workload. I was extremely disappointed with aws support and to think i am paying for support.

AMI Not found or Deleted or Made private

Relevant content