How do I troubleshoot status check failures on my Amazon EC2 instance?

Short description

Amazon EC2 uses three status checks to monitor the health of EC2 instances:

• System status checks that detect issues with an instance's underlying hardware.
• Instance status checks that check whether the instance is reachable.
• Attached Amazon Elastic Block Store (Amazon EBS) status checks that monitor whether the instance's Amazon EBS volumes are reachable and can complete I/O operations.
  Note: These status checks are for instance that are built on the AWS Nitro System only.

Important: Some status check resolutions require you to stop and start your instance. A reboot keeps the instance on the same physical host. To migrate to new hardware, you must stop and start the instance.

Resolution

First, view the instance's status check metrics to identify the status check that failed. Then, take the following actions based on the status check that failed.

Configure your instance for a stop and start

Note: When you stop and start an instance, the instance's public IP address changes. It's a best practice to use an Elastic IP address to route external traffic to your instance instead of a public IP address. If you use Amazon Route 53, then you might need to update the Route 53 DNS records when the public IP address changes. A stop and start is different from an instance reboot. For more information, see How EC2 instance stop and start works.

Before you stop and start your instance, take the following actions:

• If your instance uses an instance store, then save the volume data on the instance store to persistent storage. For example, you can save the data to EBS volumes or Amazon Simple Storage Service (Amazon S3) buckets.
  Important: Amazon EC2 deletes instance store data when you stop the instance.
• Create a snapshot of your EBS volume. If you experience issues, then you can use the snapshot to restore your instance.
• Temporarily remove the instance from its Amazon EC2 Auto Scaling group so that you don't accidentally terminate the instance when you stop it.
  Note: EC2 Auto Scaling might terminate stopped instances in an Auto Scaling group based on your scale-in protection settings. Instances that you launch with Amazon EMR, AWS CloudFormation, or AWS Elastic Beanstalk might be in an Auto Scaling group.
• Set the instance shutdown behavior to Stop to make sure that the instance doesn't terminate when you stop it.

Troubleshoot system status check failures

System status check failures occur when there are issues with the underlying infrastructure that your instance runs on.

To troubleshoot system status check failures, check the AWS Health Dashboard for service interruptions within the AWS Region where your instance is located.

If there are no outages, then there's an issue with the underlying host of the instance. To troubleshoot this issue, complete the following steps to migrate the instance to a new underlying host:

1. Stop the instance.
   Note: If the instance is stuck in the Stopping state, then force stop the instance. This action can take up to 10 minutes.
2. Start the instance.

To automatically recover instances that fail system status checks, set up automatic instance recovery.

Important: A force stop doesn't flush file system caches and can cause data loss or corruption. After a restart, run the following commands based on your operating system (OS) to check the file system for consistency errors.

Linux:

fsck

Windows:

chkdsk 

sfc /scannow	

The chkdsk command checks the file system and file system metadata of the volume for logical and physical errors. The sfc /scannow command checks file systems for corruption. For more information, see chkdsk and Using System File Checker on the Microsoft website.

Troubleshoot instance status check failures

Instance status checks identify issues with the OS, network configuration, and resource usage. To troubleshoot instance status check failures, use the following resources based on your instance OS:

• For Linux, see How do I troubleshoot status check failures for my Amazon EC2 Linux instance?
• For Windows, see How do I troubleshoot status check failures on my Amazon EC2 Windows instance?

Troubleshoot attached EBS status check failures

If your instance fails the attached EBS status check, then check the VolumeStalledIOCheck metric in Amazon CloudWatch for the affected volume. If the value is 1, then the volume has issues. Typically, Amazon EBS automatically diagnoses and recovers the volume within a few minutes. For more information, see Amazon EBS I/O characteristics and monitoring.

To validate that the volume recovered, check whether that VolumeStalledIOCheck changes to a value of 0.

If the issue persists, then stop and start the instance to migrate it to a new host. If a specific volume still has issues after the restart, then you must replace the volume. Create a snapshot of the volume, and then create a new volume from the snapshot. Use the new volume to replace the existing volume.

Related information

Troubleshoot an unreachable Amazon EC2 instance