How do I troubleshoot status check failures for my EC2 Linux instance?

Short description

Amazon EC2 uses three status checks to monitor the health of EC2 instances.

System status check

The system status check detects issues with an instance's underlying hardware. If the underlying hardware is unresponsive or unreachable because of network, hardware, or software issues, then the system status check fails.

Instance status check

An instance status check fails when you can't reach the instance. Instance status checks might fail for the following reasons:

• The operating system (OS) fails to boot
• The Amazon Elastic Block Store (Amazon EBS) volumes fail to mount correctly
• The CPU and memory are exhausted
• Kernel panic
• Network failure
• Throttling of root EBS volume parameters

Attached EBS status checks

The attached EBS status checks monitor whether the EBS volumes that are attached to an instance are reachable and can complete I/O operations. For more information, see Attached EBS status checks.

Resolution

To see whether the instance status check or system status check failed, see the instance's status check metrics.

If the system status check failed, then see Why did my EC2 Linux instance fail a system status check?

If the instance status check failed, then check the instance's system logs to see the cause of the failure. Then, choose one of the following resolution options to resolve the issue.

Important: Some of the following resolutions require you to stop and then start an instance. Data in an instance store volume is lost when you stop the instance. If your instance doesn't have EBS backed volumes, then back up your data before you stop the instance. Also, your instance's public IPv4 address can change after you stop and start the instance. To retain the same public IPv4 address, use an Elastic IP address. For more information, see Stop and start Amazon EC2 instances.

The OS fails to boot

If the system logs contain boot errors, then see How do I troubleshoot an EC2 Linux instance that failed the instance status check due to operating system issues?

The EBS volumes fail to mount correctly

A mount point failure might cause the instance status check to fail.

Example mount point failure output:

[FAILED] Failed to mount /
See 'systemctl status mnt-nvme0n1p1.mount' for details.
[DEPEND] Dependency failed for Local File Systems.

For more information on these errors, see Why does my EC2 Linux instance going into emergency mode when I try to boot it? Also, see How do I troubleshoot an EC2 Linux instance that failed the instance status check due to operating system issues?

When you change an instance type from a Xen to a Nitro-based instance, the volume mount might fail. Mount failure occurs because EBS volumes are exposed as NVMe block devices on Nitro-based instances. For example, the device names are /dev/nvme0n1 and /dev/nvme1n1. Device names that you specify in a block device mapping are renamed to NVMe device names. For example, /dev/nvme[0-26]n1.

Note: The block device driver might assign the NVMe device names in a different order than the order that you specify in the block device mapping. To avoid mount failure on Nitro-based instances, it's a best practice to use either a label or UUID for device names. For more information, see Make an Amazon EBS volume available for use.

The CPU and memory are exhausted

High CPU Utilization

If the CPUUtilization metric is at or near 100%, then the instance doesn't have enough compute capacity to run the kernel.

For T2 or T3 instances, check the Amazon CloudWatch CPU credit metrics to see if the CPU credits are at or near zero. If the CPU credits are at zero, then the CPUUtilization metric shows a saturation plateau at the baseline performance for the instance. For example, the baseline performance might be 20% or 40%. If CPU utilization is at or near 100%, or T2 or T3 instances have reached a saturation plateau, then the status check shows as failed because of resource overutilization.

To troubleshoot this issue, see How do I troubleshoot an EC2 Linux instance that fails a status check because of resource over-usage?

Block device errors, software bugs, or kernel panic can cause an unusual CPU usage spike. If CPU utilization is at 100%, then check the system logs for block device or memory issue errors, or for other unusual system errors. Then, reboot or stop and start the instance.

Out of memory

High memory pressure can cause an instance status check failure. In the following example log extract, the OS is out of memory, and the OOM-killer starts. To resolve this error, stop the process that consumes the most memory.

[115879.769795] Out of memory: kill process 20273 (httpd) score 1285879 or a child
[115879.769795] Killed process 1917 (php-cgi) vsz:467184kB, anon-rss:101196kB, file-rss:204kB

By default, EC2 instance memory and disk metrics aren't sent to CloudWatch. For more information, see Collect metrics, logs, and traces with the CloudWatch agent.

To troubleshoot and resolve the out-of-memory issue, upgrade the instance to a larger instance type. Or, add swap storage to the instance to alleviate the memory pressure. For more information, see How do I allocate memory to work as a swap file in an Amazon EC2 instance? Also, see How do I use a partition on my hard drive to allocate memory to work as swap space on an Amazon EC2 instance?

Disk full errors

If the system logs contain disk full errors, then the instance is in emergency mode because the root device is full.

Example system log:

$: sudo service apache2 restart
Error: No space left on device

$: sudo /etc/init.d/mysql restart
[....] Restarting mysql (via systemctl):
mysql.serviceError: No space left on device
         
$: df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       7.7G  7.7G     0 100% /

For more information, see How do I troubleshoot an EC2 Linux instance that fails a status check because of resource over-usage? Also, see How do I increase the size of my EBS volume if I receive an error that there's no space left on my file system?

Kernel panic

Kernel panic occurs when the kernel detects an internal fatal error during operation. If the kernel doesn't properly load, then the error occurs during the OS boot. A kernel load failure causes an instance boot to fail.

Example kernel panic error output:

Linux version
2.6.16-xenU (builder@xenbat.amazonsa) (gcc version 4.0.1 20050727 (Red Hat4.0.1-5)) #1 SMP Mon May 28 03:41:49 SAST 2007
Kernel command
line:  root=/dev/sda1 ro 4
Registering block device major 8
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(8,1)

For information, see How do I resolve the "Kernel panic - not syncing" error in my EC2 instance? Also, see How do I revert to a known stable kernel after an update blocks my EC2 instance reboot?

Network failure

Your network can fail for the following reasons:

• The cloud-init package isn't installed on the instance.
• The cloud-init package is used to update network configurations at launch.

To correct this error and install the cloud-init package on your instance, run the following commands in your terminal.

Amazon, Amazon Linux 2, Amazon Linux 2023, or RedHat OS:

sudo yum install cloud-init -y

Ubuntu or Debian OS:

sudo apt install cloud-init -y

MAC address is hardcoded in a configuration file

Hardcoded MAC addresses are in the Linux configuration files and the udev configuration files. You can find these files in the following locations:

• /etc/udev/rules.d/
• /etc/udev/rules.d/70-persistent-net.rules
• /etc/udev/rules.d/80-net-name-slot.rules

To resolve network issues that are caused by a hardcoded MAC address, remove the entries or configuration files, and then run the following command:

sudo mv /etc/udev/rules.d/70-persistent-net.rules /root/

After you move the configuration file, restart the network service to make sure that you receive a new MAC address.

The IP address is hardcoded in a network configuration file

When you create an Amazon Machine Image (AMI) from an instance with a statically configured IP address, the configuration file contains a hardcoded IP address. To correct this error, set your network interface to use DHCP.

Note: You can't update AMIs that already exist. You must set the network interface to use DHCP before you create a new AMI.

The ENA or Intel-enhanced network drivers are missing

For more information on missing Elastic Network Adapters (ENAs) or Intel-enhanced network drivers, see Enhanced networking on Amazon EC2 instances.

The network interface is automatically renamed at startup

To deactivate predictable network interface renaming, add net.ifnames=0 to the kernel command line. To use the placeholder, you must activate enhanced networking with the ENA and rebuild or update the grub configuration file.

Throttling of root EBS volume parameters

When the root EBS volume parameters are throttled, the instance might fail status checks because the instance becomes unreachable and unresponsive.

Throttling can occur when an EBS volume's I/O operations per second (IOPS) or throughput exceed the already provisioned limits. The instance might become unresponsive or unreachable because of the performance degradation from EBS volume throttling.

To resolve EBS volume throttling issues, complete the following steps:

1. Monitor and analyze CloudWatch metrics such as volume queue length, volume read/write operations, and volume read/write bytes. For more information, see How do I use CloudWatch metrics to calculate the average throughput and average number of IOPS that my EBS volume provides?
2. Stop and start the instance, or reboot the instance, to temporarily solve the issue.
3. Provision more IOPS or throughput of the EBS volume. Or, upgrade to an EBS volume type and size that's better suited to your workload. For more information, see Request Amazon EBS volume modifications.

Related information

Troubleshoot Amazon EC2 Linux instances with failed status checks

Why is my EC2 Windows instance down with a system status check failure or status check 0/2?

Why is my EC2 Windows instance down with an instance status check failure?

Types of status checks