An EC2 instance suddenly stopped working, even though we did not change anything on the server.
The website hosted on the server is no longer accessible. Also, we cannot connect to the server via SSH or with the EC2 instance connection from AWS.
When we restart the instance, the instance status is "Running", the system check is successful, but the instance status check failed.
When we try to connect to the server via SSH, we get a connection timeout on port 22.
When we try to connect to the server via the EC2 instance connection we get this error message : "Failed to connect to your instance
EC2 Instance Connect is unable to connect to your instance. Ensure your instance network settings are configured correctly for EC2 Instance Connect. For more information, see Set up EC2 Instance Connect at https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-connect-set-up.html."
We also have other servers running on the same subnet, they are running fine. Therefore, I think it is unlikely that it is due to the setting of the VPC, subnet or routing tables.
The following image shows the system log of the instance.
No error message is displayed in the whole log and the last entry of the log is always the login prompt
We have tried the following things to fix the error. However, none of them helped and after restarting the instance, the log file shown above was always displayed.
- We followed the instruction "Retrieve the system logs" on the AWS-Support site click
- We have looked at the resource monitor and all values are far below our maximum.
- We have changed the server to different and better instance types.
- We have restored several backups. All were created well before the error but it did not change anything.
- We were able to access the server using the EC2 Serial Console. We can enter username and password. Unfortunately, there is no user for the server that has a normal password. You can only connect with the private key. Therefore, the Serial EC2 console is not useful for us.