Skip to content

Windows EC2 became unresponsive and both system and instance status checks failed

-1

We experienced an issue with a Windows Server 2022 EC2 instance running on an m6i.4xlarge instance type.

During the incident:

  • The application services became unavailable.
  • RDP access stopped working.
  • Our monitoring heartbeat stopped.
  • Both EC2 status checks failed:
    • StatusCheckFailed_System = 1
    • StatusCheckFailed_Instance = 1
  • We initiated a reboot from the EC2 console, but the reboot took an unusually long time.
  • The instance recovered after approximately 13–14 minutes.

Windows Event Viewer recorded:

  • Kernel-Power Event ID 41: “The system has rebooted without cleanly shutting down first.”
  • EventLog Event ID 6008: “The previous system shutdown was unexpected.”
  • Kernel-Boot reported that the last shutdown was not successful.

We reviewed the Windows System Event Log before the outage and did not find:

  • BugCheck or blue screen events
  • WHEA hardware errors
  • Disk or storage timeout events such as 7, 11, 51, 129, or 153
  • Memory or kernel resource exhaustion events
  • ENA, NDIS, or TCP/IP errors before the outage
  • Abnormal application service termination
  • A planned restart event such as Event ID 1074

Before the failure, CPU utilization was approximately 17–24%, more than 30 GiB of physical memory was available, network utilization was low, and there were no discarded or error packets.

All operating system and monitoring metrics stopped at approximately the same time.

Windows Update components were active shortly before the incident, but there was no planned restart, update failure, rollback, or BugCheck event. A Microsoft Defender intelligence update was installed only after the instance recovered.

Could this behavior indicate an underlying EC2 host, hypervisor, or host-network impairment?

What additional AWS-side metrics or logs can be checked to distinguish between:

  1. An underlying AWS infrastructure issue
  2. A Windows OS or driver freeze
  3. A Windows Update-related issue
  4. An EBS or ENA issue

Is StatusCheckFailed_System together with StatusCheckFailed_Instance sufficient evidence that the underlying host may have been impaired?

asked 2 months ago67 views

2 Answers
0

When both system status checks and instance status checks fail simultaneously, this is a strong indicator of an underlying infrastructure issue rather than a guest OS problem. The system status check specifically monitors the AWS systems required to use your instance and detects problems with underlying host hardware, network connectivity, or power issues. When this check fails, it typically indicates issues outside your control that require AWS involvement.

The pattern you've described—sudden loss of all connectivity, monitoring metrics stopping simultaneously, both status checks failing, and the Kernel-Power Event ID 41 indicating an unexpected shutdown—is consistent with an underlying host or hypervisor impairment rather than a Windows OS freeze or software issue.

To distinguish between the different failure scenarios:

For AWS infrastructure issues:

  • Check CloudWatch metrics for any gaps in data collection during the incident window
  • Review EC2 console for any scheduled maintenance notifications or retirement notices
  • Look for patterns of StatusCheckFailed_System metrics, which specifically indicate AWS infrastructure problems
  • Check if other instances on the same host experienced similar issues (though you may not have visibility into this)

For Windows OS or driver issues:

  • Windows Event Viewer would typically show BugCheck events, driver errors, or system service failures before the incident
  • You would usually see only StatusCheckFailed_Instance (not system checks) for OS-level freezes
  • CPU, memory, or disk metrics would show abnormal patterns leading up to the failure

For EBS issues:

  • CloudWatch EBS volume metrics would show increased latency or queue depth
  • Windows Event Logs would contain disk timeout events (Event IDs 7, 11, 51, 129, 153)
  • VolumeReadOps, VolumeWriteOps, and VolumeThroughputPercentage metrics would show anomalies

For ENA network driver issues:

  • Windows Event Viewer would show NDIS or network adapter errors
  • NetworkPacketsIn/Out metrics would show drops or errors before complete failure
  • Typically wouldn't cause system status check failures

Given that you found no Windows errors preceding the incident, metrics stopped simultaneously, and both status checks failed together, this strongly suggests an AWS infrastructure impairment. The 13-14 minute recovery time is also consistent with AWS's automated recovery processes for host issues.

For this type of incident, you should open a support case through the AWS Support Center (https://console.aws.amazon.com/support/) and provide:

  • The instance ID and region
  • The exact timestamp of the incident
  • Screenshots of the status check failures
  • CloudWatch metric graphs showing the simultaneous metric gaps
  • Relevant Windows Event Viewer entries

AWS Support can access host-level logs and telemetry that aren't visible to customers, which can definitively determine if there was underlying infrastructure impairment. They can also check if the host experienced hardware issues, hypervisor problems, or network impairments that affected your instance.

To reduce the impact of future infrastructure issues, consider implementing:

  • Auto Scaling groups with health checks to automatically replace unhealthy instances
  • Multi-AZ deployments to provide redundancy
  • CloudWatch alarms on status check metrics to trigger automated recovery actions
  • AWS Systems Manager for automated instance recovery

answered 2 months ago

EXPERT

reviewed 2 months ago

0

Hello.

If it is a problem on the AWS side, you should receive an event that informs you of the EC2 infrastructure problem in the health event of your AWS account.
If you have not received this event notification, there may be a problem within the OS, not on the AWS side.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-recover.html

However, if a "StatusCheckFailed_System" error is detected, the issue might lie with AWS.
Since issues on the AWS side cannot be resolved by the user, you will need to contact AWS Support to verify the situation.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-system-instance-status-check.html

System status checks monitor the AWS systems on which your instance runs. These checks detect underlying problems with your instance that require AWS involvement to repair. When a system status check fails, you can choose to wait for AWS to fix the issue, or you can resolve it yourself. For instances backed by Amazon EBS, you can stop and start the instance yourself, which in most cases results in the instance being migrated to a new host. For instances backed by instance store (supported only for Linux instances), you can terminate and replace the instance. Note that instance store volumes are ephemeral and all data is lost when the instance is stopped.

EXPERT

answered 2 months ago

EXPERT

reviewed 2 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.