How can I regain access to my EC2 Linux instance ?

0

I have a RedHat instance I created several years ago as a DNS server, and have been using SSH access from the internet, with a key. Recently I made what I thought was a trivial change - add "LogLevel VERBOSE" to sshd_config, and I think restarted sshd. Trying to login with SSH now fails. I had also made some provisioning changes earlier, trying to enable IPv6, but the instance had been running normally all the time, for hours after I last touched networking.

I tried to login from the web console. There's a choice of EC2 Instance connect, EC2 Instance connect Endpoint, Session Manager, SSH client, and Serial console.

EC2 Instance Connect previously worked. It now says "EC2 Instance Connect is unable to connect to your instance" I created an endpoint, but EC2 Instance Connect Endpoint says "Failed to connect to your instance. Error establishing SSH connection to your instance. Try again later." It works on a new instance.

Session Manager said "No IAM". I created an IAM role, which gives a web console on a new instance I made, but on my original instance I get "SSM Agent is not online. The SSM Agent was unable to connect to a Systems Manager endpoint to register itself with the service."

Serial Console says "This instance type is not supported for the EC2 serial console."

I rebooted the instance, but that did not help. The instance is running. The system log and console snapshot show the system running. The DNS server is working normally. nmap from the internet shows: Not shown: 997 filtered ports PORT STATE SERVICE 22/tcp closed ssh 53/tcp open domain 80/tcp closed http

If I am completely unable to regain command-line access, is there a way to access the file on disk ?

adaviel
asked 2 months ago155 views
2 Answers
1
Accepted Answer

Now fixed.

I created a new instance with default settings which happened to be in the correct availability zone. I was then able to stop the original instance and detach the volume, then attach it to the new instance as /dev/sdb. In Linux on the new instance I mounted /dev/sdb2 on /mnt/sdb2 and edited /mnt/sdb2/etc/ssy/sshd_cconfig. I then unmounted /mnt/sdb2, detached the volume from the new instance, attached it to the original instance and restarted it.

However, I did not realize that the external IP address had changed, so I still could not login. EC2 connect was not working even though in theory I still had the same keypair, but I was able to repeat the detach/attach/edit process and enable password login. I was then able to login on the internal address from the new instance, and, realizing my mistake, login from the internet using my key.

Update - It occurs to me that I could have made other changes while the volume was mounted on the other instance, such as enabling password logins, or adding other SSH keys to .ssh/authorized_key etc., to let me login from older hosts that might have SSH algorithm incompatibilities.

I still have an issue with keys for EC2 connect for another question.

adaviel
answered 2 months ago
profile picture
EXPERT
reviewed 2 months ago
1

Hello.

I suspect that the problem is probably due to a change in the SSH settings.
Therefore, as described in the document below, stop the EC2 that is currently experiencing the problem, detach the root volume, and then attach it to the normal EC2 and restore the SSH settings to the original.
After that, if you attach the root volume to the original EC2 and start it, you may be able to connect via SSH.
https://repost.aws/knowledge-center/ec2-instance-boot-issues

Alternatively, you may want to try changing the EC2 settings so that you can connect via serial console.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-serial-console-prerequisites.html

profile picture
EXPERT
answered 2 months ago
profile picture
EXPERT
reviewed 2 months ago
profile pictureAWS
EXPERT
reviewed 2 months ago
  • Yes. I took a snapshot of the image, created a volume from that and attached it to another instance. Then I could see the syslog. I had mis-spelled VERBOSE. However, the other instance is in another availability zone; I was able to copy the snapshot to the other zone.

  • It might be a good idea to set the user data to use something like the "sed" command to remove "LogLevel VERBOSE". https://repost.aws/knowledge-center/execute-user-data-ec2

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions