Windows EC2 became unresponsive and both system and instance status checks failed
Dun
asked 2 months ago

Question

We experienced an issue with a Windows Server 2022 EC2 instance running on an m6i.4xlarge instance type.

During the incident:

- The application services became unavailable.
- RDP access stopped working.
- Our monitoring heartbeat stopped.
- Both EC2 status checks failed:
  - StatusCheckFailed_System = 1
  - StatusCheckFailed_Instance = 1
- We initiated a reboot from the EC2 console, but the reboot took an unusually long time.
- The instance recovered after approximately 13–14 minutes.

Windows Event Viewer recorded:

- Kernel-Power Event ID 41:
  “The system has rebooted without cleanly shutting down first.”
- EventLog Event ID 6008:
  “The previous system shutdown was unexpected.”
- Kernel-Boot reported that the last shutdown was not successful.

We reviewed the Windows System Event Log before the outage and did not find:

- BugCheck or blue screen events
- WHEA hardware errors
- Disk or storage timeout events such as 7, 11, 51, 129, or 153
- Memory or kernel resource exhaustion events
- ENA, NDIS, or TCP/IP errors before the outage
- Abnormal application service termination
- A planned restart event such as Event ID 1074

Before the failure, CPU utilization was approximately 17–24%, more than 30 GiB of physical memory was available, network utilization was low, and there were no discarded or error packets.

All operating system and monitoring metrics stopped at approximately the same time.

Windows Update components were active shortly before the incident, but there was no planned restart, update failure, rollback, or BugCheck event. A Microsoft Defender intelligence update was installed only after the instance recovered.

Could this behavior indicate an underlying EC2 host, hypervisor, or host-network impairment?

What additional AWS-side metrics or logs can be checked to distinguish between:

1. An underlying AWS infrastructure issue
2. A Windows OS or driver freeze
3. A Windows Update-related issue
4. An EBS or ENA issue

Is StatusCheckFailed_System together with StatusCheckFailed_Instance sufficient evidence that the underlying host may have been impaired?

re:Post Agent · Answer

When both system status checks and instance status checks fail simultaneously, this is a strong indicator of an underlying infrastructure issue rather than a guest OS problem. The system status check specifically monitors the AWS systems required to use your instance and detects problems with underlying host hardware, network connectivity, or power issues. When this check fails, it typically indicates issues outside your control that require AWS involvement.

The pattern you've described—sudden loss of all connectivity, monitoring metrics stopping simultaneously, both status checks failing, and the Kernel-Power Event ID 41 indicating an unexpected shutdown—is consistent with an underlying host or hypervisor impairment rather than a Windows OS freeze or software issue.

Key distinguishing factors in your case:

**Evidence pointing to infrastructure issues:**
- Both status check types failed together
- All metrics stopped at the same time (CPU, memory, network)
- No Windows errors or warnings preceded the failure
- System resources were healthy (low CPU, ample memory, normal network)
- No disk, driver, or hardware errors in Windows Event Viewer
- The instance experienced what appears to be a hard power cycle

**What this rules out:**
- A Windows OS freeze would typically show StatusCheckFailed_Instance only, not StatusCheckFailed_System
- Windows Update issues would generate specific events and wouldn't cause system status check failures
- EBS issues would show disk timeout events (Event IDs 7, 11, 51, 129, 153) in Windows logs
- ENA driver problems would show network adapter errors before failure

**Additional AWS-side investigation:**

To get more definitive information about what occurred on the AWS infrastructure side, you should open a support case through the AWS Support Center. When you do, AWS Support can:

- Review internal host health metrics and logs that aren't visible to customers
- Check for scheduled maintenance events or unscheduled host issues
- Examine hypervisor logs around the time of the incident
- Verify if there were any network infrastructure events
- Determine if the underlying hardware experienced any impairment
- Provide specific details about what triggered the system status check failure

**From your side, you can:**
- Review CloudWatch metrics for any anomalies in the minutes before the failure
- Check if AWS Personal Health Dashboard shows any events for your region and availability zone during that timeframe
- Enable detailed monitoring if not already active to capture metrics at 1-minute intervals
- Consider implementing automated recovery actions for status check failures

The 13-14 minute recovery time and the fact that the instance did recover suggests the AWS infrastructure detected and remediated the issue, which is consistent with automated host recovery processes.

Given the evidence, particularly the simultaneous failure of both status checks with no preceding Windows errors, this strongly suggests an underlying AWS infrastructure issue rather than a problem originating from the Windows OS, drivers, or applications. AWS Support will be able to provide you with specific details about what occurred on the infrastructure side and whether any preventive measures are recommended.

Windows EC2 became unresponsive and both system and instance status checks failed

Add your answer

Relevant content