Short description

Before you use a secondary IP address to connect to your instance through SSH, confirm that your instance meets the following prerequisites:

• If you launched your instance in a private subnet, then use the secondary private IP address to connect through SSH. Make sure to choose the secondary IPv4 address from the IPv4 CIDR block range of the subnet for the network interface.
• If you launched your instances in a public subnet, then allocate an Elastic IP address to your instance's secondary private IP address.
• Make sure that the operating system on your EC2 instance recognizes the secondary private IP address. For Amazon Linux, see Configure the operating system on your instance to recognize secondary private IPv4 addresses. For Ubuntu, see How can I make my secondary network interface work in my Ubuntu EC2 instance? For other Linux distributions, see the network configuration documentation for your distribution.

If your instance meets these prerequisites, then use the following steps to troubleshoot connection issues through SSH:

1. Connect through SSH with verbose messaging turned on to identify the error.
2. Review the system logs to look for errors.
3. Review the network configuration file.

Note: It's a best practice to maintain backups of your instances and data. Before you troubleshoot, create an AMI or create snapshots of your Amazon Elastic Block Store (Amazon EBS) volumes.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, make sure that you're using the most recent AWS CLI version. Also, review the general connection prerequisites before you start.

Connect through SSH with verbose messaging turned on to identify the error

For detailed instructions, see How do I troubleshoot problems connecting to my Amazon EC2 Linux instance using SSH?

Review the network configuration file and system logs to look for errors

If the issue isn't resolved, then review the instance's system logs. Use one of the following methods to access the logs and further troubleshoot on your instance.

Method 1: Connect to your instance by using a primary IP address

1.    Access your instance through SSH by using a primary IP address. After you get access, check the network status and configuration of the secondary network interface. To do this, run the following commands.

Examine the ifconfig output and make sure that the secondary network interface of your instance is up:

$ ifconfig -a

Or run the following command:

$ ip a show < network interface name> up
Example Command:
$ ip a show eth1 up

Note: In this example, the secondary network interface is eth1. Replace this with the name of your secondary network interface.

2.    Review the secondary network interface configuration file and validate all of the parameters. For example, you can check any MAC addresses, secondary IP addresses and secondary interface names. If you find any discrepancies, then edit the file and update to the correct details. Run one of the following commands:

Linux, RHEL, CentOS:

$ sudo cat /etc/sysconfig/network-scripts/ifcfg-eth1

Ubuntu

sudo cat /etc/network/interfaces.d/51-eth1.cfg  
or  
sudo cat /etc/netplan/51-eth1.yaml

3.    Run the following command to restart the network services:

$ sudo service network restart

If the issue isn't resolved, then review the system logs and authentication-related logs for the time frame when you attempted access.

Method 2: Use the Amazon EC2 Serial Console to access your instance

If you can't access your instance with a primary IP address over SSH, then use the Amazon EC2 Serial Console for Linux. The Amazon EC2 Serial Console can be used to troubleshoot supported Nitro-based and Bare Metal instance types. Before you use the Amazon EC2 Serial Console, review Amazon EC2 Serial Console for Linux instances and Configure access to the Amazon EC2 Serial Console.

Method 3: Use a rescue instance to access the network configuration file and logs

Important: Before you use this method, note the following limitations:

• If you stop an instance, then you erase any data on the instance store volumes. You must back up any data on the instance store volume that you want to keep. For more information, see Determine the root device type of your instance.
• Check if your instance is part of an Amazon EC2 Auto Scaling group. If you launch the instance with Amazon EMR, AWS CloudFormation, or AWS Elastic Beanstalk, then your instance might be part of an AWS Auto Scaling group. In this use case, if you stop the instance, then the instance might terminate. Instance termination depends on the instance scale-in protection settings for your Auto Scaling group. If your instance is part of an Auto Scaling group, then temporarily remove the instance from the Auto Scaling group before you start the resolution steps.
• If you stop and start an instance, then the instance changes its public IP address. It's a best practice to use an Elastic IP address instead of a public IP address to route external traffic to your instance.
• Check if the shutdown behavior of the instance is set to Terminate. This means that the instance is terminated if you use the shutdown or power off commands from the operating system. To avoid this, change the instance shutdown behavior.

Use the following steps to access the network configuration file with a rescue instance:

1.    Open the Amazon EC2 console.

2.    Choose Instances from the navigation pane, and then select the instance that you want to connect to.

3.    Choose Instance State, and then choose Stop Instance.

4.    Choose Stop, and then note the Instance ID.

Note: If you don't use the new Amazon EC2 experience, then select the instance that you want to connect to. Choose Actions, Instance State, Stop, and then Stop again.

5.    Detach the root EBS volumes from the stopped instance. Note the device name of the root EBS volume. The device name is required when you reattach the volume after investigation.

6.    Launch a new EC2 instance in the same Availability Zone (AZ) as the original instance. The new instance becomes your rescue instance.

Note: It's a best practice to use an Amazon Linux 2 instance as the rescue instance. This doesn't let the rescue instance boot up from the attached EBS volume because the UUID or name of the EBS volume are the same.

7.    After the rescue instance launches, choose Volumes from the navigation pane. Then select the detached root volume of the impaired instance.

Note: If your instance has AWS Marketplace codes and isn't an Amazon Linux instance, then stop the rescue instance before you attach the root EBS volume. An instance might have AWS Marketplace codes if you launched the instance from an official RHEL or CentOS AMI, for example.

8.    Choose Actions, and then choose Attach Volume.

9.    Select Instances in the navigation pane, and then select the rescue instance.

10.    Choose Instance state, and then choose Start instance.

Note: If you aren't using the new Amazon EC2 experience, select the instance that you want to connect to, and then choose Actions. Choose Instance State, and then choose Start.

11.    Connect to the rescue instance through SSH.

12.    Run the following command to verify that the EBS volume successfully attached to the rescue instance. In the following command, the volume attached is /dev/sdf.

$ lsblk

The following is an example of the command output:

NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
xvda    202:0    0   20G  0 disk 
└─xvda1 202:1    0   20G  0 part /
xvdf    202:80   0  100G  0 disk

13. Run the following commands to create a mount point directory, and then mount the attached volume to the rescue instance. In the following example, the mount point directory is /test.

$ sudo su
$ mkdir /test
$ mount /dev/xvdf1 /test

Note: When you mount the device /dev/xvdf1, if you face any issues related to duplicate UUIDs, then modify the previous command and run the command again:

$ mount -o nouuid /dev/xvdf1 /test 
$ df -h
$ cd /test

14.    Locate errors in the system logs and authentication-related logs. You can use timestamps to check logs for the times that you attempted access:

Amazon Linux, RHEL, CentOS:

$ sudo cat /test/var/log/messages

Amazon Linux, RHEL, CentOS (authentication-related issues):

$ sudo cat /test/var/log/secure

Ubuntu, Debian (System Logs):

$ sudo cat /test/var/log/syslog

Ubuntu, Debian (authentication-related issues):

$ sudo cat /test/var/log/auth.log

15.    Review the secondary network interface file given in Method 1 in this article. After you review the configurations and address any errors, unmount and then detach the EBS root volume from the rescue instance.

$ umount /test

16.    Attach the volume to the original instance. The device name is /dev/xvda.

Related information

Connect to your Linux instance using an SSH client