- Newest
- Most votes
- Most comments
In addition to the re:Post Agent's response, there is a technical clarification regarding the recommendation to "monitor CloudWatch metrics for memory utilization."
EC2 does not provide guest operating system-level metrics, such as memory or swap utilization, by default. The standard hypervisor only tracks CPU, network I/O, and disk I/O. If a Java application caused an Out of Memory (OOM) condition leading to the instance status check failure, you will not see this reflected in your default CloudWatch console.
To gain visibility into memory consumption and proactively alert on the resource exhaustion that likely caused this hang, you must install and configure the unified Amazon CloudWatch Agent on your Linux instance.
You can find the official AWS documentation and the steps to collect these internal system-level metrics here: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html
Without this agent, you will have a blind spot regarding memory leaks, making it difficult to set up the preventive alarms mentioned in the previous answer.
Based on the symptoms you described, an instance status check failure while system and EBS checks pass typically indicates an issue with the instance's software configuration, operating system, or application layer rather than underlying AWS infrastructure or storage problems.
Common causes for this specific failure pattern include:
Instance status checks can fail due to incorrect networking or startup configuration, exhausted memory, corrupted file systems, or incompatible kernels. Since your system status check passed, this rules out AWS infrastructure issues like loss of network connectivity, system power, or hardware problems on the physical host. The passing EBS check confirms your storage volumes were reachable and completing I/O operations normally.
Regarding your specific questions:
Yes, operating system hangs, memory exhaustion, network stack issues, or Java application resource problems can all cause instance status check failures. When an instance becomes unresponsive due to these issues, it may still appear "running" but cannot properly respond to health checks or serve traffic, resulting in the ALB 504 timeouts you observed.
The "rc.local is not marked executable" message is typically just an informational startup warning and unlikely to be the root cause of your instance becoming unresponsive, though it's worth ensuring your startup scripts are properly configured.
For investigation and prevention:
To identify the root cause if this recurs, monitor CloudWatch metrics for memory utilization, disk I/O, and network performance alongside CPU. The dropped network traffic you observed is a key indicator. Within the operating system, examine logs for Out of Memory (OOM) messages, kernel panics, or application-specific errors using the journalctl commands you're already running.
Your approach to reviewing previous boot logs is correct. Pay particular attention to any OOM killer messages, Java heap exhaustion errors, or network interface issues that occurred before the instance became unresponsive.
Preventive measures for production instances:
Consider implementing CloudWatch alarms for critical metrics including memory usage, status check failures, and application-specific health indicators. Configure your ALB health checks appropriately and consider using Auto Scaling to automatically replace unhealthy instances. Ensure your Java application has appropriate memory limits configured and monitor for memory leaks. Regular review of system logs and application logs can help identify issues before they cause outages.
The inability to retrieve a screenshot when the instance was unresponsive suggests the instance's operating system or graphics subsystem was not functioning properly, which aligns with an OS-level or application-level hang rather than an AWS infrastructure issue.
Sources
Status checks for Amazon EC2 instances - Amazon Elastic Compute Cloud
Understand why EC2 instance stopped | AWS re:Post
