How do I troubleshoot an Amazon EC2 Linux instance that fails a status check because of resource over-usage?

Short description

Your EC2 instance might fail its status health check because of resource usage for the following reasons:

• Your instance's CPU utilization is close to 100% and the instance didn't have enough compute capacity left for the kernel to run.
• The root device is 100% full and doesn't allow other processes to complete or begin.
• The processes that run on the instance used all its memory and don't allow the kernel to run.

To troubleshoot this issue, check the instance's metrics in Amazon CloudWatch to identify the cause of the high resource usage. Also, check the instance's system logs for errors. Then, troubleshoot on the issue that you identify.

Note: Some of the following resolution steps require you to stop and start your instance.

Resolution

Identify the root cause of the high resource usage

Check the instance's metrics

If the instance's CPUUtilization metric is at or near 100%, then reboot your instance to return the instance to a healthy status. If the problem still occurs after reboot, then the instance's CPU requirements are higher than what your instance type offers. If you use a burstable instance type, then check whether the burst credit balance is low or at 0. To troubleshoot these issues, see How do I troubleshoot high CPU utilization on an Amazon EC2 Linux instance?

Check the instance's system log for errors

Check the system log for errors.

Configure your instance for a stop and start

Note: When you stop and start an instance, the instance's public IP address changes. It's a best practice to use an Elastic IP address to route external traffic to your instance instead of a public IP address. If you use Amazon Route 53, then you might need to update the Route 53 DNS records when the public IP address changes. A stop and start is different from an instance reboot. For more information, see How EC2 instance stop and start works.

Before you stop and start your instance, take the following actions:

• If your instance uses an instance store, then save the volume data on the instance store to persistent storage. For example, you can save the data to Amazon Elastic Block Store (Amazon EBS) volumes or Amazon Simple Storage Service (Amazon S3) buckets.
  Important: Amazon EC2 deletes instance store data when you stop the instance.
• Create a snapshot of your Amazon EBS volume. If you experience issues, then you can use the snapshot to restore your instance.
• Temporarily remove the instance from its Amazon EC2 Auto Scaling group so that you don't accidentally terminate the instance when you stop it.
  Note: EC2 Auto Scaling might terminate stopped instances in an Auto Scaling group based on your scale-in protection settings. Instances that you launch with Amazon EMR, AWS CloudFormation, or AWS Elastic Beanstalk might be in an Auto Scaling group.
• Set the instance shutdown behavior to Stop to make sure that the instance doesn't terminate when you stop it.

Stop and start your instance to force the kernel to stop running processes. This is a temporary workaround to return resources to the operating system (OS). To permanently resolve resource over-usage, take the following actions to address the root cause.

Troubleshoot the "No space left on device" error

If the file system that contains a listed folder is full, then you receive an error that's similar to the following example:

"OSError: [Error 28] No space left on device '/var/lib/'"

In the preceding example, /var/lib is full.

Use the EC2 Serial Console to troubleshoot

Prerequisites: You must meet the EC2 Serial Console prerequisites.

To free up space, use the serial console to connect to instances that are built on the AWS Nitro System and supported bare metal instances. Then, delete any unneeded files.

You don't need a functional connection to connect to your instance when you use the EC2 Serial Console. However, if your instance is unreachable and you haven't already configured access to the serial console, then you can't use the EC2 Serial Console.

Use a rescue instance to troubleshoot

If you can't use the EC2 Serial Console, then complete the following steps to launch a rescue instance and remove unneeded files:

1. Launch a rescue instance in your virtual private cloud (VPC). Use the same Amazon Machine Image (AMI) and Availability Zone as the instance that failed its status check.
   Note: You can also use an existing instance that's in the same Availability Zone and uses the same AMI as the original instance.

2. Stop the original instance.

3. Detach the root EBS volume, such as /dev/xvda or /dev/sda1, from the original instance. Note the device name of your root volume.

4. Attach the volume as a /dev/sdf secondary device to the rescue instance.

5. Use SSH to connect to your rescue instance.

6. To create a mount point directory for the volume that you attached to the rescue instance, run the following command:

   sudo mkdir /rescue

   Note: Replace /rescue with your mount point directory name.

7. Run the following command to identify the correct device name:

   sudo -i
   lsblk

   Note: The device that's attached to the rescue instance might have a different device name.

8. To mount the volume to the new directory, run the following command:

   sudo mount /dev/xvdf1 /rescue

   Note: Replace dev/xvdf1 with your root volume's device name and /rescue with your mount point directory name. If you receive an error when you run the preceding command, then see Why can't I mount my Amazon EBS volume?

9. To identify the files that take up the most space, run the following command:

   du -shcm /rescue/var/lib/* |sort -n

10. To free up space, delete large files that you don't need.

11. To unmount the secondary device from your rescue instance, run the following command:

    sudo umount /rescue

    Note: Replace /rescue with your mount point directory name.
    If the unmount operation fails, then stop or reboot the rescue instance. Then, rerun the preceding command.

12. Detach the secondary volume from the rescue instance.

13. Attach the volume to the original instance as the /dev/xvda or /dev/sda1 root volume.

14. Start the instance, and then confirm that the instance is responsive.

Resize the root volume

If you still don't have enough storage available, then complete the following steps to resize the root volume:

1. Request a volume size modification.
2. After you receive the updated volume, follow steps 1-8 of Use a rescue instance to troubleshoot to launch a rescue instance.
3. Use the rescue instance to extend the Linux file system.

Troubleshoot the "Out of memory" error

If your instance is out of memory, then you receive the following error message:

"Out of memory: kill process"

If the instance is out of memory, then the kernel doesn't have enough memory to run and Amazon EC2 ends other processes to free memory. To troubleshoot this issue, see How do I troubleshoot high memory usage issues on my EC2 Linux instance? For more information, see Out of memory: kill process.

Note: If you can't connect to your instance to troubleshoot, then follow steps 1-8 of Use a rescue instance to troubleshoot to launch a rescue instance.

Troubleshoot "page allocation failure" errors

If the kernel memory allocator fails the allocation request, then you receive a "page allocation failure" error.

To troubleshoot this issue, it's a best practice to upgrade the instance to a larger instance type. Or, for instances that use ephemeral instance store volumes, use a swap file or hard drive partition to add swap storage to the instance. This configuration alleviates the memory pressure.

The instance uses swap space when the RAM is full. You can use swap space for instances that have a small amount of RAM.

Note: Swap space isn't a replacement for more RAM. Because swap space is on the instance's hard drive, performance is slower compared to RAM. For more or faster memory, you must increase the instance size.

Use atop to investigate and prevent resource issues

Use the atop monitoring tool to identify resource usage patterns and processes that might cause issues before they cause system failures. The atop tool helps you monitor and troubleshoot the system on an ongoing basis.

Note: The atop tool logs data only after you install it. You can't retrieve historical performance data from before you installed the atop tool.

Use the atop tool to check the following resources:

• Check historical data in /var/log/atop/ to analyze your system and processes when the failure occurred.
• Identify processes that consume a large amount of resources.
• Check the memory, CPU, and disk usage patterns that led up to the failure.

Related information

How do I troubleshoot status check failures for my Amazon EC2 Linux instance?

What steps do I need to take before I change the instance type of my EC2 Linux instance?