How do I get my Amazon ECS tasks that use the Amazon EC2 launch type to pass the Application Load Balancer health check?

10 minute read

I want to troubleshoot and resolve issues with Application Load Balancer health checks for Amazon Elastic Container Service (Amazon ECS) tasks that run on my Amazon Elastic Compute Cloud (Amazon EC2) instances.

Short description

When your Amazon ECS task fails the load balancer health check, you receive one of the following errors from your Amazon ECS service event message:

"(service AWS-service) (port 8080) is unhealthy in (target-group arn:aws:elasticloadbalancing:us-east-1:111111111111:targetgroup/aws-targetgroup/123456789) due to (reason Health checks failed with these codes: [502 or 504]) or (request timeout)"
"(service AWS-Service) (port 8080) is unhealthy in target-group tf-20190411170 due to (reason Health checks failed)"
"(service AWS-Service) (instance i-1234567890abcdefg) (port 443) is unhealthy in (target-group arn:aws:elasticloadbalancing:us-east-1:111111111111:targetgroup/aws-targetgroup/123456789) due to (reason Health checks failed)"

You might receive the following error from your Amazon ECS task console:

"Task failed ELB health checks in (target-group arn:aws:elasticloadbalancing:us-east-1:111111111111:targetgroup/aws-targetgroup/123456789)"

For failed container health checks, see How do I troubleshoot container health check failures for Amazon ECS tasks?

To determine why your Amazon ECS task stopped, see Viewing Amazon ECS stopped task errors and Why has my Amazon ECS task stopped?

Resolution

To resolve Application Load Balancer health check failures for your Amazon ECS tasks on Amazon EC2 instances, complete one or more of the following actions.

Configure different security groups

It's a best practice to configure different security groups for your load balancers and container instances.

Confirm that your security groups include the following rules:

The security group for your load balancer allows outbound traffic to your container instances on the registered port and the health check port.
The security group for your container instances allows inbound traffic on the task host port range from the load balancer's security group.

For steps to add or update security group rules, see Configure security group rules. For the recommended security group rules for an Application Load Balancer, see Security groups for your Application Load Balancer.

Turn on the Availability Zone for your load balancer

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, confirm that you're using the most recent AWS CLI version.

When you configure an Availability Zone for your load balancer, Elastic Load Balancing creates a load balancer node in the Availability Zone. If you register targets in the Availability Zone but don't turn on the Availability Zone, then the registered targets don't receive traffic. For more information, see Availability Zone subnets.

To determine the Availability Zones that your load balancer is configured for, complete the following steps:

Open the Amazon EC2 console.
In the navigation pane, under Load Balancing, choose Load balancers.
Select the load balancer that you're using for your Amazon ECS service.
On the Description tab, view the Availability Zones.

Or, run the following describe-load-balancers AWS CLI command:

aws elbv2 describe-load-balancers --load-balancer-arns EXAMPLE-ALB-ARN --query 'LoadBalancers[*].AvailabilityZones[].{Subnet:SubnetId}'

Note: Replace EXAMPLE-ALB-ARN with your Application Load Balancer's ARN.

To determine the Availability Zones that your container instances are configured for, complete the following steps:

Open the Amazon EC2 console.
In the navigation pane, under Auto Scaling, choose Auto Scaling groups.
Select the container instance Auto Scaling group associated with your cluster.
On the Details tab, under Network, verify that the Availability Zones match the Availability Zones for your load balancer.

Or, run the following describe-auto-scaling-groups AWS CLI command:

aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names EXAMPLE-ASG-NAME --query 'AutoScalingGroups[*].{Subnets:VPCZoneIdentifier}' --output text

Note: Replace EXAMPLE-ASG-NAME with your Auto Scaling group's name.

To modify your cluster's Availability Zones, complete the following steps:

Open the AWS CloudFormation console.
Select your cluster's CloudFormation stack.
Choose Update.
On the Specify stack details page, update your Subnet IDs configuration.

To determine the Availability Zones that your task is configured for, complete the following steps:

Open the Amazon ECS console.
In the navigation pane, choose Clusters, and then select the cluster that contains your service.
On the Services tab, in the Service Name column, select the service that you want to check.
Choose the Configuration and Networking tab.
Under Network configuration, view the configured subnets.
Open the Amazon Virtual Private Cloud (Amazon VPC) console to view additional information that isn't available in the Amazon ECS console.
Run the following describe-services AWS CLI command to verify that your subnets' Availability Zones match your load balancer's Availability Zones:

aws ecs describe-services --cluster EXAMPLE-CLUSTER-NAME --service EXAMPLE-SERVICE-NAME --query 'services[*].deployments[].networkConfiguration[].awsvpcConfiguration.{Subnets:subnets}'

Note: Replace EXAMPLE-CLUSTER-NAME with your cluster's name. Replace EXAMPLE-SERVICE-NAME with your service's name.

You can't use the Amazon ECS console to change the subnet configuration of an Amazon ECS service. Instead, run the AWS CLI update-service command.

Configure your Network ACL to allow traffic between subnets

The subnets for your load balancer and your container instance or task network interface might be different.

Confirm that your network ACLs include the following rules:

The network ACL for your load balancer's subnets allows inbound traffic on the ephemeral ports (1024–65535) and the listener port.
The network ACL allows outbound traffic on the health check port and ephemeral ports.
The network ACL for your container instance or task network interface subnets (awsvpc) allows inbound traffic on the health check port.
The network ACL allows outbound traffic on the ephemeral ports.

For steps to add or update network ACL rules, see Create a network ACL for your VPC. For more information about network ACLs, see Control subnet traffic with network access control lists.

Check the health check settings of your target group

To check that you correctly configured the health check settings for your target group, complete the following steps:

Open the Amazon EC2 console.
In the navigation pane, under Load Balancing, choose Target Groups.
Choose the name of your target group to open its details page.
Important: Use a new target group. Because Amazon ECS automatically registers and deregisters the ECS task with the target group, don't manually add targets to the target group.
On the Health checks tab, choose Edit.
Confirm the following settings:
For Port, confirm that you correctly configured the field. If you choose Override, then confirm that the port matches the task host port.
For Path, confirm that you correctly configured the field.
For Timeout, confirm that the response timeout value is correct. If the value is lower than the time required for a response, then the health check fails.
Choose Save changes.

Check the status and configuration of the application in your ECS container

Confirm that the application responds to your load balancer health check

Check the following:

Confirm that you correctly configured the ping port and the health check path for your target group.
Monitor the CPU and memory utilization metrics for your Amazon ECS service. For steps to view these metrics, see Viewing Amazon ECS metrics. If your application is slow or times out, increase the task resource quotas, scale out the service, optimize your application, or use a larger instance type.
Set a minimum health check grace period so that the service scheduler ignores the health checks for a predefined time period after you initiate a task. Your Amazon ECS task might require a longer health check grace period to allow the application to start up completely and begin accepting incoming requests. For steps to update the health check grace period, see Update Amazon ECS service parameters.
Check your application logs for application errors. For more information, see Send Amazon ECS logs to CloudWatch.

Confirm that the application returns the correct status code

When the load balancer sends an HTTP GET request to the health check path, the application in your ECS container returns the default 200 OK status code. If you receive a non-HTTP error message, then your application isn't listening to the HTTP traffic. You might receive an HTTP status code that's different from what you specified in the Matcher setting. If you receive another status code, then your application is listening to the HTTP traffic but isn't returning a status code for a healthy target.

Note: If you use an Application Load Balancer, then you can update the Matcher setting to a status code other than 200. For more information, see Health checks for Application Load Balancer target groups.

To confirm that the application in your Amazon ECS container returns the correct status code, complete the following steps:

Use SSH, Session Manager, a capability of AWS Systems Manager, or Amazon EC2 Instance Connect to connect to your container instance.
(Optional) Run the following command for your operating system to install curl.
Amazon Linux and other RPM-based distributions:
```
sudo yum -y install curl
```
Debian-based systems, such as Ubuntu:
```
sudo apt-get install curl
```
Run the following command to get the container ID:
```
docker ps
```
Note: The port for the local listener is displayed in the command's output under PORTS at the end of the sequence.
If you use the BRIDGE network mode, then run the following docker inspect command to get the container's IP address:
```
IPADDR=$(docker inspect --format='{{.NetworkSettings.IPAddress}}' 112233445566)
```
Note: The container's IP address is saved in IPADDR. Replace 112233445566 with the container ID from the docker ps command's output. If you use awsvpc, then use the task IP address assigned to the task network interface. If you use the HOST network mode, then use the IP address of the host container instance that the task is exposed through.
To get the status code, run the following curl command that includes IPADDR and the local listener's port:
```
curl -I http://${IPADDR}:8080/health
```
Note: Replace 8080 with your listener's port.

Check the status of your container instance

If you receive the following event message from your Amazon ECS service event:

"(service AWS-Service) (instance i-1234567890abcdefg) (port 443) is unhealthy in (target-group arn:aws:elasticloadbalancing:us-east-1:111111111111:targetgroup/aws-targetgroup/123456789) due to (reason Health checks failed)"

Check the status of your container instance. If your instance fails the system status checks, then stop the instance. After the instance stops, start the instance.

Temporarily activate the Application Load Balancer access logs

Temporarily activate the Application Load Balancer access logs to check for the following issues:

Determine whether the Application Load Balancer is sending health checks to the correct path or port and whether the targets are responding correctly.
Analyze the HTTP status codes that the targets return to identify application-level issues, such as misconfigured routes or server-side errors.
Check that the health checks reached the target to determine whether there are network-related issues.
Determine whether the response time exceeds the configured health check timeout.

For steps to activate access logs, see Enable access logs for your Application Load Balancer.