Why does my CloudWatch UnhealthyHostCount metric show unhealthy targets when Application Load Balancer shows all targets as healthy?

Short description

Application Load Balancer deploys multiple nodes across your Availability Zones, and each node independently performs health checks against your registered targets. If at least one Application Load Balancer node reports the target as healthy, then the Amazon Elastic Compute Cloud (Amazon EC2) console displays a target as healthy. If at least one node reports the target as unhealthy, then the UnhealthyHostCount metric in the CloudWatch console  with the Maximum statistic shows that the target is unhealthy.

Resolution

If you use the Maximum statistic with the CloudWatch UnhealthyHostCount metric, then the count of unhealthy targets can vary between the Amazon EC2 console and the UnhealthyHostCount metric.

Understand how CloudWatch metrics aggregate health check data

It's a best practice to use CloudWatch to monitor for non-zero UnHealthyHostCount values with the Minimum statistic. Create an alarm for the Minimum statistic for a series of multiple non-zero values. You can also create an alarm for Average or Maximum statistics to investigate any non-zero occurrences.

The UnhealthyHostCount and HealthyHostCount metric values depend on the statistic that you select in CloudWatch. Review the following statistics:

• Review the Maximum statistic to see the highest count of unhealthy or healthy hosts reported by any single Application Load Balancer node during the time period. This statistic shows when an Application Load Balancer node is experiencing issues when it tries to reach your targets.
• Review the Minimum statistic to see the lowest count that an Application Load Balancer node reports. This statistic shows the health status across all nodes and detects when any node of your loaed balance considers a target as unhealthy.
• Review the Average statistic to see the mean value across all Application Load Balancer nodes that can mask issues that affect individual nodes.

Note: To check whether health status varies across Application Load Balancer nodes, view the HealthyHostCount metric in the CloudWatch console with both Maximum and Minimum statistics. A gap between these values shows differences in health status.

Troubleshoot unhealthy host counts

If your UnhealthyHostCount metric is elevated for more than a few minutes, then identify root causes in your Application Load Balancers or endpoints. Common root causes for unhealthy targets include Security group misconfiguration and Network Access Control List (ACL) misconfiguration.

Verify which Application Load Balancer nodes are reporting unhealthy targets

To identify the Application Load Balancer nodes that experience issues, check the UnhealthyHostCount metric with the AvailabilityZone dimension:

1. Open the CloudWatch console.
2. Choose Metrics, and then choose ApplicationELB.
3. Add the UnhealthyHostCount metric with dimensions for LoadBalancer, AvailabilityZone, and TargetGroup.
4. Check for non-zero values across Availability Zones to identify the Application Load Balancer nodes that have unhealthy targets.

Test health check endpoints manually

To simulate Application Load Balancer health checks, send requests directly to your target's health check path from sources in each Application Load Balancer subnet. Complete the following steps:

1. To test connectivity to the target, run the telnet command from an Amazon EC2 instance in the same Amazon Virtual Private Cloud (Amazon VPC):

   telnet TARGET_IP TARGET_PORT 

   Note: Replace TARGET_IP with your target IP address and TARGET_PORT with your target port number.
2. To simulate the health check request, run the following command:

   curl -v http://TARGET_IP:TARGET_PORT/HEALTH_CHECK_PATH 

   Note: Replace TARGET_IP with your target IP address**, TARGET_PORT** with your target port number, and HEALTH_CHECK_PATH with your health check path.
3. To do an HTTPS health checks, run the following command:

   curl -vk https://TARGET_IP:TARGET_PORT/HEALTH_CHECK_PATH  

   Note: Replace TARGET_IP with your target IP address and TARGET_PORT with your target port number, and HEALTH_CHECK_PATH with your health check path.
4. Verify that the target returns the expected HTTP status code that match your health check Matcher configuration. If responses are slow or intermittent, then investigate application-level issues.

Review target application logs

Review your target application logs to identify failed or slow health check responses that correlate with the time periods when CloudWatch reports unhealthy hosts. Take the following actions:

• Review HTTP requests to your health check path.
• Check the response times for health check requests.
• Review error messages or exceptions during health check processing.
• Check if you have resource constraints during health check failures, including CPU, memory, and disk I/O.

Related information

Health checks for Application Load Balancer target groups

Monitor your Application Load Balancers

CloudWatch metrics for your Application Load Balancer