Why can't I boot my Amazon EC2 Linux instance?

Short description

The Amazon EC2 Linux instance fails to boot, and then shows an error message in the Amazon EC2 Linux console. This error means that the instance failed to load the Security-Enhanced Linux (SELinux) policy. Reasons for the error include misconfigured SELinux settings and corrupted or missing SELinux policy files.

Resolution

To recover an instance from an SELinux policy freeze, choose one of the following options:

• Restore the instance's root volume from a recent snapshot.
• Use a rescue instance to recreate or modify the SELinux policy configuration.

Restore the instance's root volume from a recent snapshot

If you have an Amazon Elastic Block Store (Amazon EBS) root volume snapshot, then you can restore the root volume to its previous working state. For more information, see Replace the root volume for an Amazon EC2 instance without stopping it. Note: In this method, you lose data that's added after you create the snapshot.

Use a rescue instance to recreate or modify the SELinux policy configuration

To use a rescue instance to recreate or modify the SELinux configuration, complete the following steps:

1. Launch a rescue or temporary instance. Use the Amazon Machine Image (AMI) of the nonfunctioning instance to launch a new EC2 Linux instance in your virtual private cloud (VPC). Make sure that your new instance is in the same Availability Zone as the nonfunctioning instance. The new instance becomes your rescue instance. You can also use an existing instance that's in the same Availability Zone as your nonfunctioning instance.
2. Detach the Amazon Elastic Block Store (Amazon EBS) root volume from your nonfunctioning instance. Note the device name (/dev/xvda or /dev/sda1) and the volume ID (start with vol-) for later use.
3. Attach the EBS volume to the rescue instance as a secondary device, for example /dev/sdf.
4. Use Secure Shell (SSH) or AWS Systems Manager (SSM) Session Manager to connect to the rescue instance. To identify the root partition device name, run the lsblk command:

   sudo -i
   lsblk -o +PARTLABEL,LABEL,SERIAL | sed 's/vol/vol-/'

   Note: The root partition contains the label / and the partlabel Linux. If there are multiple entries, then use the volume ID to differentiate the root volume of the problematic instance.

Mount a root file system

Create a temporary mount point (for example, /mnt/rescue). To mount the root file system from the attached volume, run the following command:

mkdir -p /mnt/rescue
mount -o nouuid /dev/device-name /mnt/rescue

Note: Replace device-name with the device name of the root partition that you identified in the previous step.

To use chroot to change the root directory, run the following commands:

for i in proc sys dev run; do mount --bind /$i /mnt/rescue/$i; done
chroot /mnt/rescue

Note: In the example, the /dev, /proc, /sys, and /run directories are bind-mounted from the original root file system. This configuration allows processes that run inside the chroot environment to access these system directories.

Resolve the SELinux policy configuration

To recreate your SELinux policy, run the following command:

semodule -B

To deactivate SELinux, run the following command:

sed -i -E 's/^SELINUX=(permissive|enforcing)/SELINUX=disabled/' /etc/selinux/config

Unmount and reattach the volume from the rescue instance

1. To exit the chroot environment and unmount the volume, run the following command:

   exit
   umount /mnt/rescue/{proc,sys,dev,run,}

   Note: If the /mnt/rescue: target is busy, then use the -l option (for example, umount -l /mnt/rescue) to force the detach.
2. Detach the volume from the rescue instance. Then, reattach the volume to the original instance as the root volume (device name either /dev/xvda or /dev/sda1).
3. Start the original instance and then verify that the boot issue resolved.

Back up and test your SELinux policy
It's a best practice for your SELinux policy files to complete the following steps:

1. Back up the working SELinux policy before you modify it.
   To back up your policy, run the following tar command:

    tar -czvf selinux-policy-store-backup.tar.gz /etc/selinux/targeted/policy/

2. Test the changes in a non-production environment.

Related information

System cannot boot: Failed to load SELinux policy, freezing on the Red Hat website

SELinux Users and Administrators Guide Red Hat Documentation on the Red Hat website