How do I resolve the "Kernel panic - not syncing" error in my EC2 instance?

Short description

The "Kernel panic - not syncing" error occurs when the device or address doesn't exist. To resolve this issue, launch a temporary instance, and then attach the faulty root disk as a secondary drive to perform diagnostics.

Important: Before you stop and start your instance, take the following actions:

• Create a backup of your Amazon Elastic Block Store (Amazon EBS) volume.
  Note: If your instance is instance store-backed or has instance store volumes that contain data, then Amazon EC2 deletes the data when you stop the instance.
• Temporarily remove the instance from its Amazon EC2 Auto Scaling group when you're completing the resolution steps.
  Note: If you stop an instance that's in an Amazon EC2 Auto Scaling group, then you might terminate the instance based on your scale-in protection settings. Instances that you launch with Amazon EMR, AWS CloudFormation, or AWS Elastic Beanstalk might be in an Auto Scaling group.
• Set the instance shutdown behavior to Stop to make sure that the instances don't terminate when you stop them.

Note: When you stop and start an instance, the instance's public IP address changes. It's a best practice to use an Elastic IP address to route external traffic to your instance instead of a public IP address.

For more information, see What happens when you stop an instance.

Resolution

Note: The following resolution steps are only for Amazon Linux 2, Amazon Linux 2023, Fedora 16 and later, and Red Hat Enterprise Linux (RHEL) 7 and later.

To attach the root disk to a temporary instance, complete the following steps:

1. Get the volume ID and device name for the original instance's root volume.

2. Stop the original instance.

3. Launch a temporary instance from an Amazon Machine Image (AMI) with the same Linux operating system (OS) version in the same Availability Zone.

4. Detach the root volume from the original instance and attach it to the temporary instance as a secondary volume. Note the volume device name.

5. Use the SSH key pair to connect to the temporary instance.

6. To change to the root user, run the following command:

   sudo su

7. To identify the block device name and partition, run the following command from the temporary instance:

   lsblk

   Example output:

   NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
   xvda    202:0    0    8G  0 disk
   └─xvda1 202:1    0    8G  0 part /
   xvdf    202:80   0  101G  0 disk
   └─xvdf1 202:81   0  101G  0 part
   xvdg    202:80   0   10G  0 disk

   This example uses a XEN instance with blkfront drivers.

8. If you use a partitioned volume, then run the following command to mount the /dev/xvdf1 partition instead of the /dev/xvdf device:

   mount -o nouuid  /dev/xvdf1 /mnt

   Note: Both /dev/xvda and /dev/xvdf are partitioned volumes, but /dev/xvdg isn't.
   If you use an instance built on the AWS Nitro System, then the volume device name is similar to /dev/nvme[0-26]n1. To mount the partition on the /mnt directory, run the following command:

   mount -o nouuid  /dev/nvme1n1p1 /mnt

   Note: Replace nvme1n1p1 with the block device name that you identified in step 7. For more information, see Device names for volumes on Amazon EC2 instances.

9. To create a chroot environment in the /mnt directory, run the following command:

   for i in dev proc sys run; do mount -o bind /$i /mnt/$i; done; chroot /mnt

   This example bind-mounts the /dev, /proc, /sys, and /run directories from the original root file system. This configuration allows processes that run inside the chroot environment to access the system directories.

10. To create a backup of the initramfs in the / directory, run the following command:

    for file in /boot/initramfs-*.img; do cp "${file}" "/$(basename "$file")_$(date +%Y%m%d)"; done

11. To list the default kernel, run the following command:

    grubby --default-kernel

    Example output:

    /boot/vmlinuz-5.15.156-102.160.amzn2.x86_64

    The preceding output lists the kernels that boot at startup.

12. To list the kernels and initramfs in the boot directory, run the following command:

     ls -lh /boot/vmlinuz* && ls -lh /boot/initr*

    Example output:

    -rwxr-xr-x. 1 root root 9.7M Apr 23 20:37 /boot/vmlinuz-5.10.215-203.850.amzn2.x86_64-rwxr-xr-x. 1 root root 9.9M Apr 23 17:00 /boot/vmlinuz-5.15.156-102.160.amzn2.x86_64
    -rw-------. 1 root root 12M May 3 23:45 /boot/initramfs-5.10.215-203.850.amzn2.x86_64.img
    -rw-------. 1 root root 9.8M May 14 08:03 /boot/initramfs-5.15.156-102.160.amzn2.x86_64.img

    Note the vmlinuz kernel files that have corresponding initramfs files.

13. To rebuild the initramfs, run the following command:

    dracut --force --verbose /boot/initramfs-kernelVersion.img kernelVersion

    Note: Replace kernelVersion with the latest kernel version.

14. To determine whether the instance is booting on UEFI or BIOS, run the following command:

    boot_mode=$(ls /sys/firmware/efi/efivars >/dev/null 2>&1 && echo "EFI" || echo "BIOS"); echo "Boot mode detected: $boot_mode"

15. Update the grub configuration. If your instance boots on BIOS, then run the following command:

    grub2-mkconfig -o /boot/grub2/grub.cfg

    Note: When you run the preceding command, you might receive the "device-mapper: reload ioctl on osprober-linux-xvda2 (253:0) failed: Device or resource busy Command failed" error message. To resolve this issue, add the GRUB_DISABLE_OS_PROBER=true parameter to the /etc/default/grub file, and then run the command again.
    If your instance boots on UEFI, then run the following commands based on your OS.
    UEFI:
    Amazon Linux 2 and Amazon Linux 2023:

    grub2-mkconfig -o /boot/efi/EFI/amzn/grub.cfg

    Fedora 16+:

    grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg

    Red Hat 7+:

    grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

16. To exit and detach the volume, run the following command:

    exit; umount -fl /mnt

17. Detach the secondary volume from the temporary instance and attach it to the original instance as the root device. Use the device name that you noted in step 4.

18. Connect to the original instance.