How do I troubleshoot the connection errors that I get with the Application Load Balancers over HTTP?

Short description

The Application Load Balancer maintains two separate TCP connections:

• The TCP connection that it establishes with the client.
• The TCP connection that it establishes with the target.

The Application Load Balancer might have several concurrent TCP connections opened with a single target at any time. The client sends a request to the load balancer over the frontend TCP connection. The Application Load Balancer forwards the request to any target in the target group over the backend TCP connections. In other words, there isn't a one-to-one relationship between client TCP connections and TCP connections with the targets. For more information, see HTTP connections.
Note: An Application Load Balancer is a Layer 7 load balancer that supports both HTTP and HTTPs listeners.

Resolution

The following section provides troubleshooting tips for general TCP connection issues that might occur when the Application Load Balancer handles regular HTTP requests.

Identify which TCP connection is problematic (the frontend or the backend connection)

Perform the following simple telnet or netcat test for this sample configuration:

 telnet <ALB DNS name> <ALB listener port>

If the client connects to the Application Load Balancer, you see the following output:

Connected to <ALB DNS>

Frontend connectivity issues with the client

If you face timeout issues when you run the frontend TCP connectivity test, check the following:

• Is the client allowed in the Application Load Balancer's security group?
• Is the client allowed in the network access control list (network ACL) for the Application Load Balancer's subnet?
• Are there any intermediate devices, such as firewalls, between the client and the Application Load Balancer?

If the issue persists, check for the following:

• If your Application Load Balancer is public, then check if the load balancer has a route out to the internet.
• If the Application Load Balancer is private, it must have a route to an AWS Virtual Private Network (AWS VPN) or AWS Direct Connect.

Verify if you're using the correct subnet

Intermittent timeouts occur when the Application Load Balancer is configured incorrectly and placed in the wrong subnet.

For example, consider a public Application Load Balancer that lives in two Availability Zones:

us-east-1a --> subnet A
us-east-1b --> subnet B

• If connections only succeed a portion of the time, make sure both subnet A and subnet B have a route out to the internet.
• If subnet B is private (no route to an internet gateway), the connections behave differently. The connections routed to the Application Load Balancer's IP in subnet B time out while the connections routed to subnet A succeed. Make sure that both the subnets are configured with a route out to the internet.

Backend connectivity issues with the target

A successful connection to the Application Load Balancer itself doesn't indicate that the backend target is reachable.

To test connectivity with the backend, run the following CURL command to the Application Load Balancer's DNS:

curl <ALB DNS:listener port> -v  

With the -v flag activated, you get the following output:

* Trying <ALB public IP>:80...
* Connected to <ALB DNS> (public IP) port <listener port>

The previous section of the output indicates that a TCP connection was successfully established from the client to the Application Load Balancer.

The following section of the output shows the client request that was sent to the Application Load Balancer:

> GET / HTTP/1.1
> Host: <ALB DNS name>
> User-Agent: curl/8.4.0
> Accept: /

The following section of the output shows the response coming back from the target. The target is an Apache server with a simple hello world output:

< HTTP/1.1 200 OK  
...  
<h1>Hello World</h1>  
< Server: Apache

When the connection to the backend fails due to TCP timeouts between the Application Load Balancer and the target, you get a 504 error. The error code indicates that there was a connection issue. The output might be similar to this:

< HTTP/1.1 504 Gateway Timeout
< Server: awselb/2.0

If connection to the backend failed because the target sent a TCP Reset (TCP RST), you get this type of response:

< HTTP/1.1 502 Bad Gateway
< Server: awselb/2.0

For more information on HTTP error codes, see The load balancer generates an HTTP error.

To fix a backend connection issue, follow these troubleshooting tips:

• Check the security groups and network ACLs to make sure that your target is reachable.
• Make sure that your target is listening on the target group port.
• Check if your setup has firewall software that's blocking the Application Load Balancer from sending requests to the target.
• Test connectivity from another Amazon Elastic Compute Cloud (Amazon EC2) instance within the same subnet as one of the Application Load Balancer nodes.