How do I troubleshoot failed health checks for Application Load Balancers?

7 minute read
0

I want to find out why the targets that are registered to my Application Load Balancer aren't healthy.

Resolution

To troubleshoot failed health checks for your Application Load Balancer, find the reason code and description of your issue. Then, complete the following tasks to resolve your issue.

Find the reason code and description of your issue

Use the resource map instead of the target group's console to view the load balancer's resources and identify unhealthy targets. The resource map shows all the resources of the Application Load Balancer on a single page.

Note: If the target HTTP response of the Application Load Balancer isn't the expected response, then check your application's response. Make sure that the application sends the correct response to the load balancer.

Resolve the issue based on the reason code

Based on the health check reason code that you find, complete the following tasks to resolve the issue.

Elb.InitialHealthChecking

Before a target can receive requests from the load balancer, the target must pass initial health checks. Wait for your target to pass the initial health checks, and then recheck its health status.

Elb.RegistrationInProgress

The load balancer starts routing requests to the target as soon as the registration process completes and the target passes the initial health checks.

Target.DeregistrationInProgress

When you deregister a target, the load balancer no longer sends requests to the target. Elastic Load Balancing waits 300 seconds before it completes the deregistration. However, you can use the Amazon EC2 console or the AWS Command Line Interface (AWS CLI) to update the delay value. For more information, see Deregistration delay.

Note: If you receive errors when you run AWS CLI commands, then see Troubleshoot AWS CLI errors. Also, make sure that you're using the most recent AWS CLI version.

Target.FailedHealthChecks

To resolve this issue, complete the following tasks:

  • Confirm that your application is running. Run the service command to check the status of services on Linux targets. For Windows targets, check the Services tab of Windows Task Manager. If the service is stopped, then start the service. If the service isn't recognized, then check whether the service is installed.

  • Verify that the target is listening for traffic on the health check port. You can run the ss command on Linux targets to verify the ports that your server is listening on. For Windows targets, you can run the netstat command.

  • Launch or use an existing instance in the same Amazon Virtual Private Cloud (VPC) as the target instance. Make sure that you have access to this instance. Or, If the target is publicly accessible, send a health check request directly to the target's public IP address.

    Use the following CURL command:

    $curl -vkso /dev/null <HealthCheck_protocol>://<Target_IP>:<HealthCheck_port>/<HealthCheck_path> 

    Note: Use this method to bypass the Application Load Balancer and verify if the target is responding correctly to the load balancer's health check requests.

  • Verify that your application responds to the load balancer's health check requests. The following example shows a typical health check request from the Application Load Balancer that your targets must return with a valid HTTP response:

    GET / HTTP/1.1Host: 10.0.0.1:80
    Connection: close
    User-Agent: ELB-HealthChecker/2.0
    Accept-Encoding: gzip, compressed

    Note: In the preceding example, the Host header value contains the private IP address of the target, followed by the health check port. The User-agent is set to ELB-HealthChecker/2.0. The line terminator for message-header fields is the sequence CRLF, and the header terminates at the first empty line followed by a CRLF. To receive the health check requests, you might need to add a default virtual host to your web server configuration.

  • Verify that your application is listening on the correct network interface if you attached multiple interfaces to your target. For more information, see Target type.

  • Confirm that the target provides a server certificate and a key in the format that's specified in the Elastic Load Balancing security policy. Also, check that the target supports the matching ciphers and the protocol that the load balancer provides to establish the TLS handshake.

Target.InvalidState

If the target is an Amazon EC2 instance, then use the Amazon EC2 console to verify that the instance is running. If the instance isn't running, then manually start the instance.

Target.IpUnusable

If your target type is ip, then don't choose an IP address that a load balancer is already using.

Target.NotInUse

To resolve this issue, complete the following tasks:

  • Check the target group to verify that it's configured to receive traffic from the load balancer.
  • Make sure that the target's Availability Zone is turned on for the load balancer.

Target.NotRegistered

Make sure that the target is registered to the target group.

Target.ResponseCodeMismatch

To resolve this issue, complete the following tasks:

  • Review your health check configuration to verify the success codes that the load balancer is expecting to receive. By default, the success code is 200, but you can specify values between 200 and 499. Review your web server access logs to see if the expected success codes are being returned.
  • Verify that the ping path is valid. Be sure to specify a valid URI. The default is /.

To change your success code value or ping path, see Modify the health check settings of a target group.

Target.Timeout

If you can connect, then the target page might not respond before the health check timeout period. Most web servers, such as NGINX and IIS, let you log how long the server takes to respond. For more information, see Configuring logging on the NGINX website and Configure logging in IIS on the Microsoft website.

If your health check requests take longer than the configured timeout, then complete the following tasks:

If you can't connect, then complete the following tasks:

  • Use the health check port and health check protocol to verify that the security group that's associated with the target allows traffic from the load balancer. You can add a rule to the security group to allow all traffic from the load balancer security group. Also, the security group for your load balancer must allow traffic to the targets.
  • Confirm that the network access control list (network ACL) that's associated with your target's subnets allows inbound traffic on the health check port. The network ACL must also allow outbound traffic on the ephemeral ports (1024-65535).
  • Verify that the network ACL that's associated with your nodes' subnets allows inbound traffic on the ephemeral ports. The network ACL must also allow outbound traffic on the health check and ephemeral ports.
  • Check that the OS-level firewalls on the target allow health check traffic in and out.
  • Verify that the route table for the target's subnets contains an entry that allows health check traffic back to the load balancer.
  • Confirm that your target's memory and CPU utilization are within acceptable capacity. If your memory or CPU utilization is too high, then add targets or increase the capacity of your Auto Scaling group. If your target is an EC2 instance, then change the instance to a larger instance type.

Related information

Troubleshoot your Application Load Balancers

AWS OFFICIAL
AWS OFFICIALUpdated 2 months ago