Why is my Amazon EC2 Linux instance unresponsive?

Short description

EC2 instances become unresponsive for the following reasons:

• CPU usage is consistently at or near 100%, and doesn't allow the kernel to perform typical operating system (OS) operations.
• The instance ran out of memory, so the Linux Out of Memory (OOM) Manager terminated processes such as the database, web server, or SSH service.
  Note: EC2 instances don't allocate swap space by default.
• The root device is 100% full and doesn't allow other processes to complete or begin, and you receive "No space left on device" errors.
• Your instance can't communicate with other hosts because of network issues.
• I/O operation issues cause read and write instructions, such as the creation of temporary files, reads from system libraries, or databases, to get stuck.

The preceding issues can also accumulate. For example, you run out of memory so the OOM manager terminates a key process. Then, a process that relies on the stopped process starts a higher number of CPU cycles. If the process is disk-related, then the cycle might also use all available Amazon Elastic Block Store (Amazon EBS) volume space. Or, the issue might transfer to an instance that expects communication from the unresponsive instance.

To troubleshoot the unresponsive instance, identify the root cause of the issue, and then take troubleshooting actions based on that issue.

Resolution

Identify the root cause of the issue

Monitor CPU usage with CloudWatch

Use the Amazon CloudWatch agent to see trends in your CPU usage and identify whether the issue is ongoing, spontaneous, or recurring.

Use system monitoring tools

If you have multiple services, then you must identify the service overuses resources. Use tools such as sar or atop and atopsar to get historical usage statistics about the processes that run on your instance. For information about how to configure the tools, see How do I configure the ATOP monitoring and SAR monitoring tools for my EC2 instance that runs Amazon Linux, Red Hat Enterprise Linux (RHEL), CentOS, or Ubuntu?

Check system logs

Check your systems logs for errors that relate to high resource usage to identify system or application issues.

Run the following command to view kernel messages:

dmesg 

Also, check the /var/log/syslog or /var/log/messages files for system messages.

Review command history

Check the ~/.bash_history file for a history of commands to identify issues such as an unexpected script that's running in the background.

Check for high memory usage

Run the following command to check memory usage:

free -h

In the output, check the used and free columns to identify your instance's memory usage. For more information about the free command, see Dissecting the free command: What the Linux sysadmin needs to know on the Red Hat website.

Check for high storage usage

If your root volume storage usage is high, then your device might be full. To identify this issue, check the instance console logs.

Troubleshoot high CPU usage

Identify the process that uses high CPU

Use the pidstat or ps command to get more detailed information about processes, such as whether the process is a system or user process. For more information about the commands, see pidstat and ps on the Linux manual page.

Check for scheduled jobs

Check whether scheduled jobs or cron jobs that ran on the instance caused the high CPU usage. First, check the CloudWatch metrics for your instance to identify the timestamp of when CPUUtilization was high.

Then, run the following command to list the cron jobs:

sudo crontab -lsudo cat /etc/crontab
sudo cat crontab -l 

Note: The preceding command checks the crontab configuration for the root user. To check for a specific user, add -u username to the command, and then replace username with the user's username.

In the command output, check the jobs that ran at the time when CPU usage was high.

Check the following logs:

• /var/log/messages/var/log/syslog
• /var/log/dmesg
• /var/log/cron.log

Use the grep command to filter entries in the logs for specific cron jobs and check for errors. For more information about the command, see grep on the Linux manual website.

Check network traffic

High network traffic might cause high CPU usage, especially if the instance manages a large volume of network requests.

Run the following command to monitor network traffic:

iftop

Use the command output to identify opportunities to optimize your network configuration. Make sure that your configuration doesn't throttle network traffic and that the traffic is within the correct bandwidth for your instance type. For more information, see Linux interface analytics on-demand with iftop on the Red Hat website. If network traffic is consistently high, then it's a best practice to upgrade your instance type to a larger size.

Check disk I/O

High disk I/O might cause high CPU usage. To identify processes that cause high I/O, run the following command:

iostat

For more information, see I/O reporting from the Linux command line on the Red Hat website.

To end a process that you don't require, run the kill command. For more information, see kill on the Linux manual website.

To further troubleshoot high CPU usage, see How do I troubleshoot high CPU utilization on an Amazon EC2 Linux instance?

Troubleshoot high memory usage

To troubleshoot high memory usage, see How do I troubleshoot high memory usage issues on my EC2 Linux instance?

Troubleshoot "no space left on device" issues

To troubleshoot high root device usage, use the EC2 Serial console to connect to your instance and delete unneeded files. You can use the EC2 Serial console for instances that are built on the AWS Nitro System and supported bare metal instances. You don't need a working connection to connect to your instance when you use the EC2 Serial Console. However, you must adhere to the prerequisites for the EC2 Serial console.

If your instance is unreachable and you haven't already configured access to the serial console, then you can't use the EC2 Serial Console. In this scenario, use SSH to connect to your instance, and then increase the size of the Amazon EBS volume.

If you can't use the serial console or SSH, then launch a rescue instance and use it to delete unneeded files. For instructions, see Troubleshoot the "No space left on device" error on How do I troubleshoot an EC2 Linux instance that fails a status check because of resource over-usage?

Optimize your code

If your application causes high CPU usage, then identify and eliminate performance bottlenecks in your code. Use tools such as perf or strace to identify code issues. For instructions, see Recording and analyzing performance profiles with perf and How do I use strace to trace system calls made by a command? on the Red Hat website.

Upgrade your instance

If your processes consistently use a large volume of resources for valid reasons, such as a high user intake, then increase your instance size. For information about requirements, see What steps do I need to take before I change the instance type of my EC2 Linux instance?

Use the AWS Compute Optimizer to identify the correct instance type and size for your configuration requirements. You can also use Amazon EC2 Auto Scaling to scale horizontally.

Configure Linux audit rules

For more visibility into user commands and configuration changes, configure the Linux Audit system to track changes.

Update your configuration to avoid future resource overusage

Before you deploy a new application in production, create a test environment. Use the environment to benchmark the application and determine the required compute, memory, EBS volume capacity, and network traffic. Then, deploy your application based on the benchmark requirements. For best practices to build for fault tolerance, see Design interactions in a distributed system to prevent failures.

Check the configuration files, database queries, and code that run on your instance for high performance. If your application is database-heavy, then it's a best practice to use caching to reduce the number of queries to the database.

Make sure that your software is up to date with the latest security patches and bug fixes. Outdated software might cause performance issues that cause high CPU usage.

If your application receives high traffic volume, then use a load balancer to distribute the traffic across multiple EC2 instances. A load balancer reduces the CPU usage on individual instances.

Continually monitor the resources usage of your instances. Create CloudWatch alarms to automatically identify instances at risk of resource overusage.