Why doesn't my Amazon EC2 Linux instance boot after I changed it to a Nitro-based instance?

Short description

An EC2 Linux instance might not boot after you change it to a Nitro-based instance because of the following reasons:

• You didn't activate the enaSupport enhanced networking attribute on the instance.
• You didn't install the Elastic Network Adapter (ENA) driver on the instance.
• You didn't install the NVMe driver on the instance. Or, you didn't load the NVMe module in the initramfs image of the instance.
• You used a device name to mount the file systems in the /etc/fstab file at boot instead of a UUID.
  Note: Amazon Elastic Block Store (Amazon EBS) volumes appear as NVMe devices to Nitro-based instances. This configuration changes the device names.

To resolve boot issues, change the instance type back to a Xen instance. Then, use the AWSSupport-MigrateXenToNitroLinux runbook to automatically prepare your instance for migration. Or, manually check and fix the missing prerequisites. Finally, change the instance type to a Nitro instance.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Change the instance type back to Xen

Important: To change instance types, you must stop and start the instance.

Configure your instance for a stop and start

Note: When you stop and start an instance, the instance's public IP address changes. It's a best practice to use an Elastic IP address to route external traffic to your instance instead of a public IP address. If you use Amazon Route 53, then you might need to update the Route 53 DNS records when the public IP address changes.

Before you stop and start your instance, take the following actions:

• If your instance uses an instance store, then save the volume data on the instance store to persistent storage. For example, you can save the data to Amazon EBS volumes or Amazon Simple Storage Service (Amazon S3) buckets.
  Important: Amazon EC2 deletes instance store data when you stop the instance.
• Create a snapshot of your EBS volume. If you experience issues, then you can use the snapshot to restore your instance.
• Temporarily remove the instance from its Amazon EC2 Auto Scaling group so that you don't accidentally terminate the instance when you stop it.
  Note: EC2 Auto Scaling might terminate stopped instances in an Auto Scaling group based on your scale-in protection settings. Instances that you launch with Amazon EMR, AWS CloudFormation, or AWS Elastic Beanstalk might be in an Auto Scaling group.
• Set the instance shutdown behavior to Stop to make sure that the instances don't terminate when you stop them.

Change the instance type

Change the instance back to the Xen instance type to make sure that the instance can boot.

Automatically prepare your instance for migration

To automatically update your instance to meet the prerequisites to migrate to Nitro, run the AWSSupport-MigrateXenToNitroLinux runbook.

Check prerequisites before you migrate

Run the nitro_check_script.sh script to check prerequisites before you change the instance type to Nitro. To access the script, see nitro_check_script.sh on the GitHub website. For information about what the script checks, see NitroInstanceChecks on the GitHub website.

To download the nitro_check_script.sh script, run the following command:

sudo wget https://raw.githubusercontent.com/awslabs/aws-support-tools/refs/heads/master/EC2/NitroInstanceChecks/nitro_check_script.sh -O /tmp/nitro_check_script.sh

To give your environment permission to run the script, run the following command:

sudo chmod +x /tmp/nitro_check_script.sh 

To run the script with root permissions, run the following command:

sudo bash /tmp/nitro_check_script.sh

Note: You must run the script as a root user or with sudo. Otherwise, you receive the "This script must be run as root" error message.

Based on the script output, take the following actions.

Verify that you activated enhanced networking and installed the ENA driver

Test whether you activated enhanced networking on your instance. If enhanced networking isn't activated, then activate enhanced networking.

To check whether you installed the ENA driver, run the following command:

sudo modinfo ena |grep -i '^version:' || echo "ENA module not available, try modprobe ena"

If you didn't install the ENA driver, then install the latest driver. For instructions, see Linux kernel driver for Elastic Network Adapter (ENA) family on the GitHub website.

Install the NVMe driver

To access NVMe volumes, you must install the NVMe drivers.

Check whether you used a device name for mount points in the /etc/fstab file

Complete the following steps:

1. To check whether you used a UUID, run the following command:

   cat /etc/fstab |grep -iP 'uuid|dev/'

   Example output:

   /dev/nvme0n1p1 / xfs defaults,noatime 1 1

   Note: The example output uses a device name instead of a UUID.

2. If you didn't use a UUID, then run the following command to retrieve the UUID for the device name:

   lsblk -f device_name |tail -n 1 |awk '{print $4}'

   Note: Replace device_name with your device name.

3. (Optional) Before you edit the /etc/fstab file, it's a best practice to run the following command to make a copy of the file:

   sudo cp /etc/fstab /etc/fstab-`date +%d%m%Y`

4. Run the following command to open the /etc/fstab file:

   cat /etc/fstab

5. In the file, replace the device name with the UUID.
   Example file:

   /dev/nvme0n1p1 / xfs defaults,noatime 1 1

   Note: Replace /dev/nvme0n1p1 with the UUID. Make sure that you replace all device names in the /etc/fstab file with the UUID.

6. To verify that you correctly updated the file, rerun the following command:

   cat /etc/fstab

   In the output, check whether UUID shows the UUID.
   Example output:

   UUID=123456-1234-1234-acbd-1234abc1234 / xfs defaults,noatime 1 1

Change the instance type back to Nitro

Change the instance to a Nitro-based instance.

Note: If you upgrade to the latest Nitro instance and use an outdated kernel or driver, then the instance might fail its status check. As a result, you receive boot failures and "kernel panic" errors, or your instance becomes unreachable. To troubleshoot this issue, see How do I resolve the "Kernel panic - not syncing" error in my EC2 instance?