How do I troubleshoot Application Load Balancer HTTP 502 errors?

Resolution

Identify the source of the HTTP 502 errors

For a list of error causes, see HTTP 502: Bad gateway. The source of the error can be your target or your load balancer.

To identify the source, check your load balancer's Amazon CloudWatch metrics. If the HTTPCode_ELB_502_Count metric has data points, then your load balancer is the source of the error. If the HTTPCode_Target_5XX_Count metric has data points, then your target is the source of the error.

Or, check your load balancer's access logs. If the elb_status_code is 502 and the target_status_code is -, then your load balancer is the source of the error. If both elb_status_code and target_status_code are 502, then your target is the source of the error.

To determine the cause of the error, filter the access logs by elb_status_code = "502" and target_status_code. For requests that fail to an AWS Lambda function, check the Lambda error reason codes in the error_reason field of the load balancer's access logs.

Troubleshoot HTTP 502 errors

The load balancer tries to establish a connection, but receives a TCP RST from the target

If the load balancer can't establish a three-way handshake with the target, then you receive a TCP reset (TCP RST) message. In this scenario, the load balancer can't forward the user request to the target.

Check the TargetConnectionErrorCount metric for data points that show that the target rejects connections from the load balancer with a "TCP RST".

In the access logs, check whether the request_processing_time, target_processing_time, and response_processing_time fields are set to -1. If the values are -1, then the load balancer can't send the request to the target.

To resolve this issue, make sure that the target allows TCP connections from the load balancer. For example, check the target's operating system (OS) and firewall configuration.

When the load balancer tries to connect, it receives an unexpected response from the target, such as "ICMP Destination unreachable (Host unreachable)"

If you receive the "ICMP Destination unreachable" response, then check your load balancer's virtual private cloud (VPC) configuration. The security groups and network access control lists (network ACLs) must allow TCP connections from the load balancer.

The target closes the connection with a TCP RST or a TCP FIN message when the load balancer has an outstanding request to the target

If keep-alive timeout on the target is shorter than idle timeout on the load balancer, then the target closes the connection too early. To resolve this issue, make sure that the value for keep-alive timeout is longer than idle timeout value.

The target response is malformed or contains HTTP headers that aren't valid

To troubleshoot target response issues, perform a packet capture on the target for the timeframe when the issue occurred.

To perform a packet capture for Windows, see Download Wireshark application on the Wireshark website.

To perform a packet capture for Linux, run the following command:

sudo tcpdump -i any -w filename.pcap

Important: If the target has a large amount of traffic, then the Packet Capture (PCAP) file that tcpdump generates might affect your disk space. A large amount of traffic can also affect your target's services.

To use tcpdump to test packet capture samples, see How do I troubleshoot network performance issues between Amazon Elastic Compute Cloud (Amazon EC2) Linux or Windows instances in a VPC and an on-premises host over the internet gateway?

In the packet capture, make sure that the HTTP response complies with the HTTP specifications (RFC 2616 and related). For information about the required syntax, see HTTP messages on the Datatracker website.

The load balancer encounters an SSL handshake error when it connects to a target

If there are SSL handshake issues, then the load balancer can't forward the request to the target. This issue occurs even if the TCP connection to the target's HTTPS listener is successful.

If the target group uses the HTTPS protocol, then perform a packet capture on the target for the timeframe of the issue. Use the information in the packet capture to check TCP and TLS handshake details. Make sure that your server uses a TLS cipher suite that your load balancer's security policy supports.

The deregistration delay period passed for a request that a deregistered target manages

To check whether this issue occurred, check your AWS CloudTrail Event history for the DeregisterTargets action during the timeframe of the issue. If the target deregistered too early, then you receive an "HTTP 502" error message. To resolve this issue, increase the deregistration delay so that the lengthy operations can complete.

Troubleshoot HTTP 502 errors when the target is a Lambda function

The response body exceeds 1 MB

If the error_reason field in the load balancer access log is LambdaResponseTooLarge, then the issue is the response body size. You also see data points for the LambdaUserError metric.

To troubleshoot the issue, update your Lambda code and add error handling logic so that the function can address responses that exceed 1 MB. For example, update the code to upload larger responses to an Amazon Simple Storage Service (Amazon S3) bucket. Then, create a response with the download URL for the Amazon S3 object instead of the full response.

The Lambda function didn't respond before the configured timeout

If the error_reason field in the load balancer access log is LambdaUnhandled, then the issue is that the target didn't respond. You also see data points for the LambdaUserError metric.

To resolve this issue, increase the Lambda timeout time, or update your Lambda code so that it runs within the configured timeout.

The Lambda function returned an error, or Lambda throttled the function

If you receive an error message from the Lambda function, then check the Lambda function logs for more information about the issue.

To check whether Lambda throttled the function, check the Throttles metric for a data point. If Lambda throttled the function, then implement retry with backoff logic. If you still encounter throttling issues, then request a quota increase for concurrent executions in Lambda.

For more information, see Understanding AWS Lambda invoke throttling limits.