How do I troubleshoot Network Load Balancer health check failures for Amazon ECS tasks on Fargate?

Short description

For HTTP and HTTPS health checks, see How do I troubleshoot Application Load Balancer health check failures for Amazon ECS tasks on Fargate?

When your Amazon ECS tasks fail a Network Load Balancer health check, you receive errors in your service event messages that are similar to the following examples:

• "Health checks failed error - (service AWS-service) (port 80) is unhealthy in (target-group arn:aws:elasticloadbalancing:us-east-1:111111111111:targetgroup/aws-targetgroup/123456789) due to (reason Health checks failed)"
• "Target is in an Availability Zone that is not turned on for the load balancer error - (service AWS-service) (port 80) is unhealthy in (target-group arn:aws:elasticloadbalancing:us-east-1:111111111111:targetgroup/aws-targetgroup/123456789) due to (reason Target is in an Availability Zone that is not enabled for the load balancer)"
• "Health checks requests getting timed out - (service AWS-service) (port 8443) is unhealthy in (target-group arn:aws:elasticloadbalancing:us-east-1:111111111111:targetgroup/aws-targetgroup/123456789) due to (reason Request timed out)."

If you receive container health check failures, then see How do I troubleshoot the container health check failures for Amazon ECS tasks?

If your Amazon ECS tasks are stopped, then see Viewing Amazon ECS stopped task errors.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Health checks failed

To troubleshoot load balancer health check failures on your Fargate tasks, take the following actions.

Check the connectivity between your load balancer and Amazon ECS task

To allow the load balancer to perform health checks on your Amazon ECS tasks, confirm the following configurations:

• If your container is mapped to port 80, then your task security group allows inbound traffic on port 80.
• The application can run on the port that's mapped to the container instance in the task definition.
• The elastic network interface security group allows traffic on the Amazon Virtual Private Cloud (Amazon VPC) CIDR range. For more information, see Target security groups.
• The Network Load Balancer security group rules allow outbound traffic to the Amazon ECS service security group on the required container port.
• The network access control lists (network ACL) of the network interface subnets for your task allow inbound traffic on the health check port.
• The network ACL allows outbound traffic on the ephemeral ports.

Confirm that your tasks correctly respond to manual checks within your Amazon VPC connection

Confirm that the Amazon Elastic Compute Cloud (Amazon EC2) instance tasks within your VPC correctly respond to manual checks.

Note: You can either create a cluster for the Amazon EC2 launch type, or launch a new Amazon EC2 instance. If you don't want to launch an Amazon EC2 instance, then you can use ECS Exec.

To check your task response, use SSH to connect to an EC2 instance within your Amazon VPC connection. Then, run one of the following commands to test your target group configuration.

HTTP health checks:

curl -Iv http://example-task-private-ip:example-port/healthcheck_path

Note: Replace example-task-private-ip with your task IP address, and example-port with your port. The command's output shows the successful status codes in the 200-399 range for HTTP health check configurations on the target group.

Example output:

HTTP/1.1 200 OK

TCP health checks that don't use SSL with the targets:

nc -z -v -w10 example-task-private-ip example-port

Note: Replace example-task-private-ip with your task IP address, and example-port with your port.

Example output:

nc -z -v -w10 10.x.x.x 80Connection to 10.x.x.x port 80 [tcp/http] succeeded!

TCP health checks that require SSL for backend health checks:

nc -z -v -w10 --ssl example-task-private-ip example-port

Note: Replace example-task-private-ip with your task IP address, and example-port with your port.

Example output:

nc -z -v -w10 10.x.x.x 443Connection to 10.x.x.x port 443 [tcp/https] succeeded!

Check the status and configuration of the application in your Amazon ECS container instance

Take the following actions:

• Check that you correctly configured the ping port and the health check path for your target group.
• Monitor your service's CPU and memory usage metrics for high resource consumption that causes the application not to respond to health check requests.

If your task needs a longer health check grace period, then run the following update-service AWS CLI command to increase healthCheckGracePeriodSeconds:

aws ecs update-service --cluster example-cluster --service example-service --region example-region --health-check-grace-period-seconds example-value --force-new-deployment

Note: Replace example-cluster with your cluster name, example-service with your service name, example-region with your AWS Region, and example-value with your health check grace period in seconds.

Check whether targets in your target group are failing health checks because they take longer to respond than the default timeout period. To resolve this issue, run the following modify-target-group command to increase the target group timeout seconds:

aws elbv2 modify-target-group --target-group-arn Target-Group-ARN --health-check-timeout-seconds Timeout-Value 

Note: Replace Target-Group-ARN with your target group ARN and Timeout-Value with your group timeout value in seconds.

Check your application logs for application errors.

Make sure that the response code that your application sends on HealthCheckPath matches the response code in your target group's health check parameters. If you configured access logging on your application, then use the ELB-HealthChecker/2.0 keyword to check the logged response. If you're using Amazon CloudWatch Logs, then use CloudWatch Logs Insights to run the following query:

fields @timestamp, @message| sort @timestamp desc
| filter @message like /ELB-HealthChecker/

The query's output shows the health check requests that the Network Load Balancer sends, the response code, and errors.

Target is in an Availability Zone that isn't activated for the load balancer

If you register targets in an Availability Zone, then you must activate that Availability Zone for the registered targets to receive traffic.

Important: You can't deactivate Availability Zones for a Network Load Balancer after you create the load balancer. However, you can activate additional Availability Zones.

To identify the Availability Zones that your load balancer is configured for, run the following describe-load-balancers command:

aws elbv2 describe-load-balancers --load-balancer-arn example-arn-load-balancer --region example-region --query "LoadBalancers[].AvailabilityZones[].ZoneName"

Note: Replace example-arn-load-balancer with your load balancer ARN and example-region with your Region.

To identify the Availability Zones that your Fargate task is configured for, run the following describe-services command:

aws ecs describe-services --cluster example-cluster-name --services example-service-name --region example-region --query "services[].networkConfiguration.awsvpcConfiguration.subnets"

Note: Replace example-cluster-name with your cluster name, example-service-name with your service name, and example-region with your Region. The command's output shows you the IDs for the subnets in your service.

To identify the Availability Zones of your task's subnets, run the following describe-subnets command:

aws ec2 describe-subnets --subnet-ids example-subnet-ids --region example-region --query "Subnets[].AvailabilityZone"

Note: Replace example-subnet-ids with your subnet ID and example-region with your Region. The command's output shows you the Availability Zones that your service is configured for.

To change the subnet configuration of an Amazon ECS service, run the following update-service command:

aws ecs update-service --cluster cluster-name --service service-name --network-configuration "awsvpcConfiguration={subnets=[subnet-xxxxx,subnet-yyyyy]}"

Note: Replace cluster-name with your cluster name, and service-name with your service name.

Backend dependency

If the application's health check path communicates with an upstream or downstream service, such as a database, then make sure that the services are available. Issues with the services can result in a health check failure.

Amazon ECS tasks might be in the Unhealthy status for multiple reasons. If the preceding resolution doesn't resolve your issue, then see Troubleshooting service load balancers in Amazon ECS.

Related information

Health checks for Network Load Balancer target groups

Network configuration