How do I troubleshoot status check failures for my Amazon EC2 Linux instance?

Short description

Amazon EC2 uses three status checks to monitor the health of EC2 instances:

• System status checks that detect issues with an instance's underlying hardware.
• Instance status checks that check whether the instance is reachable.
  Note: Instance status check failures occur when there are issues in the operating system (OS).
• Attached Amazon Elastic Block Store (Amazon EBS) status checks that monitor whether the instance's Amazon EBS volumes are reachable and can complete I/O operations.

Resolution

To identify the status check that failed, view the instance's status check metrics.

To troubleshoot system status check or attached EBS status check failures, see How do I troubleshoot status check failures on my Amazon EC2 instance?

If the instance status check failed, then check the instance's system logs to see the cause of the failure. Then, take the following actions based on the issue that you encounter.

Important: Some of the following resolutions require you to stop and start the instance.

Configure your instance for a stop and start

Note: When you stop and start an instance, the instance's public IP address changes. It's a best practice to use an Elastic IP address to route external traffic to your instance instead of a public IP address. If you use Amazon Route 53, then you might need to update the Route 53 DNS records when the public IP address changes. A stop and start is different from an instance reboot. For more information, see How EC2 instance stop and start works.

Before you stop and start your instance, take the following actions:

• If your instance uses an instance store, then save the volume data on the instance store to persistent storage. For example, you can save the data to Amazon EBS volumes or Amazon Simple Storage Service (Amazon S3) buckets.
  Important: Amazon EC2 deletes instance store data when you stop the instance.
• Create a snapshot of your EBS volume. If you experience issues, then you can use the snapshot to restore your instance.
• Temporarily remove the instance from its Amazon EC2 Auto Scaling group so that you don't accidentally terminate the instance when you stop it.
  Note: EC2 Auto Scaling might terminate stopped instances in an Auto Scaling group based on your scale-in protection settings. Instances that you launch with Amazon EMR, AWS CloudFormation, or AWS Elastic Beanstalk might be in an Auto Scaling group.
• Set the instance shutdown behavior to Stop to make sure that the instance doesn't terminate when you stop it.

Troubleshoot an OS that fails to boot

If the system logs contain boot errors, then there are OS issues. To troubleshoot this issue, see How do I troubleshoot an EC2 Linux instance that failed the instance status check because of OS issues?

Troubleshoot EBS volumes that didn't mount correctly

If a mount point failure caused the instance status check to fail, then you receive an output similar to the following example:

[FAILED] Failed to mount /See 'systemctl status mnt-nvme0n1p1.mount' for details.
[DEPEND] Dependency failed for Local File Systems.

To troubleshoot this error, see Troubleshoot "Failed to mount" or "Dependency failed" on How do I troubleshoot an EC2 Linux instance that failed the instance status check because of OS issues? Also, see Dependency failed errors on Why does my EC2 Linux instance go into emergency mode when I try to boot it?

If you change from a Xen-based instance type to an instance that's built on the AWS Nitro system, then the volumes might fail to mount. Nitro instances expose EBS volumes as NVMe block devices, such as /dev/nvme0n1 or /dev/nvme1n1. Device names that you specify in a block device mapping are automatically renamed to the /dev/nvme[0-26]n1 format.

Note: The block device driver might assign the NVMe device names in a different order.

To prevent mount failures, use the volume's filesystem label or UUID in /etc/fstab instead of the volume's device names. For configuration steps, see Make an Amazon EBS volume available for use.

Troubleshoot high CPU and memory

High CPU usage

If the CPUUtilization metric in Amazon CloudWatch is at or near 100%, then the instance doesn't have enough compute capacity to run the kernel.

For T2 or T3 instances, also check the CPU credit metrics. If CPU credits are at zero, then CPUUtilization stops at the baseline performance level for the instance type. For example, if baseline performance is 20% or 40%, then the instance doesn't exceed that level of CPU usage.

To troubleshoot high CPU usage issues, see How do I troubleshoot high CPU utilization on an Amazon EC2 Linux instance?

High memory

If your memory pressure is high, then you see the following error message in the logs:

"[115879.769795] Out of memory: kill process 20273 (httpd) score 1285879 or a child[115879.769795] Killed process 1917 (php-cgi) vsz:467184kB, anon-rss:101196kB, file-rss:204kB"

To resolve this issue, stop the process that consumes the most memory. If you require all running processes, then upgrade the instance to a larger instance type. Or, add swap storage to the instance or partition your hard drive to alleviate the memory pressure.

For more troubleshooting steps, see How do I troubleshoot high memory usage issues on my EC2 Linux instance?

Troubleshoot a full disk

If the root device is full, then the instance goes into emergency mode and you receive a system log output similar to the following example:

$: sudo service apache2 restart
Error: No space left on device

$: sudo /etc/init.d/mysql restart
[....] Restarting mysql (via systemctl):
mysql.service
Error: No space left on device
         
$: df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       7.7G  7.7G     0 100% /

To troubleshoot this issue, increase the size of your volume. Or, see "Troubleshoot the "No space left on device" error" on How do I troubleshoot an EC2 Linux instance that fails a status check because of resource over-usage?

Troubleshoot kernel panic errors

Kernel panic occurs if the kernel detects an internal error during the operation. If the kernel doesn't load, then the instance can't boot and you receive a system log output similar to the following example:

Linux version2.6.16-xenU (builder@xenbat.amazonsa) (gcc version 4.0.1 20050727 (Red Hat4.0.1-5)) #1 SMP Mon May 28 03:41:49 SAST 2007
Kernel command
line:  root=/dev/sda1 ro 4
Registering block device major 8
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(8,1)

To troubleshoot this issue, see How do I resolve the "Kernel panic - not syncing" error in my EC2 instance? Or, revert to a known stable kernel.

Troubleshoot network failures

To identify and troubleshoot network failures, see How do I troubleshoot OS-level network configuration issues on my Amazon EC2 Linux instance?

Troubleshoot throttled root EBS volume performance

If the EBS volume exceeds its limits for input/output operations per second (IOPS) or throughput, then Amazon EC2 throttles the volume. As a result, the instance fails its status checks and becomes unresponsive.

Monitor and analyze CloudWatch metrics for Amazon EBS to understand the expected volume IOPS or throughput.

Note: To identify a throttled volume, check for IOPS or throughput metrics that stay the same over time. For more information, see Monitor I/O characteristics using CloudWatch.

To temporarily resolve the issue, reboot or stop and start your instance to end the processes that use IOPS and throughput. For a long-term solution, upgrade to an EBS volume type and size that's better suited to your workload.

Related information

Troubleshoot Amazon EC2 Linux instances with failed status checks

How do I troubleshoot status check failures on my Amazon EC2 Windows instance?