Skip to content

How do I troubleshoot 504 errors that I receive when I use an Application Load Balancer?

5 minute read
8

I receive an "ELB_504" error in my Application Load Balancer access logs or Amazon CloudWatch metrics. Or, I receive an HTTP 504 error when I connect to my service through an Application Load Balancer.

Resolution

An HTTP 504: Gateway timeout error occurs when a gateway or proxy times out. To resolve this issue, take the following actions.

Check whether your load balancer allows traffic with registered targets

Check the TargetConnectionErrorCount metric in CloudWatch with the Sum statistic. If you see positive data points instead of 0, then there are connection issues between the load balancer and the target.

To resolve these issues, check the network security groups that are associated with the load balancer and the backend targets. Make sure that the security groups allow traffic between the load balancer and targets in both directions on the traffic and health check ports. Confirm that the subnet's network access control list (network ACL) allows traffic from the targets to the load balancer nodes on the ephemeral ports (1024-65535).

Note: It's a best practice to use specific security group rules for your Application Load Balancer.

Check your load balancer metrics

To determine the source of the 504 error, check the CloudWatch metrics for your Application Load Balancer. If there isn't HTTPCode_ELB_504_Count metric data, then your application servers returned the 504 errors, not the load balancer. Check whether the maximum value for the TargetResponseTime metric frequently reaches the load balancer's idle timeout, as this issue can cause 504 errors.

Increase your resource availability

To identify high resource usage, check CPU and memory utilization metrics in your targets. For Amazon Elastic Compute Cloud (Amazon EC2) instances, check the CPUUtilization metric. Amazon EC2 instances don't send memory metrics to CloudWatch by default. However, you can use CloudWatch Agent to send a custom memory metric to CloudWatch.

For Amazon Elastic Container Service (Amazon ECS) tasks, check the CPUUtilization and MemoryUtilization metrics. If the value for either is 1 (100%), then the task becomes unresponsive.

For AWS Lambda functions, check the Duration metric. If the Duration lasts longer than the load balancer's idle timeout value, then you receive a "Gateway timeout" error.

If your targets have high CPU usage, then they might become unresponsive. To resolve this issue, take the following actions:

Update your application's code to be more efficient when it responds to HTTP requests. Make sure that the application doesn't take more time to respond than the configured idle timeout period. By default, the idle timeout for an Application Load Balancer is 60 seconds. If needed, then increase the idle timeout of your load balancer.

Note: It's a best practice to increase the idle timeout of the Application Load Balancer only if the target has time-consuming compute operations or external dependencies. Otherwise, it's a best practice to optimize your resource usage in the targets instead.

Scale your targets based on your demand

For Amazon EC2, use Amazon EC2 Auto Scaling groups with a scaling policy for traffic spikes. For Amazon ECS, use cluster auto scaling.

Note: When you invoke a Lambda function, the function automatically scales.

Check your external dependencies

An application that uses a decoupled or microservices architecture that relies on external dependencies, such as databases or APIs, might affect the target response time.

Check the following external dependencies for issues:

  • For Amazon Relational Database Service (Amazon RDS) databases, check the ReadLatency, WriteLatency, and DatabaseConnections metrics in CloudWatch.
  • For Amazon Simple Queue Service (Amazon SQS) queues, check the ApproximateAgeOfOldestMessage and NumberOfMessagesDelayed metrics in CloudWatch.
  • For Amazon Simple Storage Service (Amazon S3) buckets, check the FirstByteLatency, TotalRequestLatency, 4xxErrors, and 5xxErrors metrics in CloudWatch.
  • For Amazon Cognito authentication services, check the TokenRefreshSuccesses metric in CloudWatch, and check for ThrottlingException errors.

To determine the source of a performance bottleneck, take the following actions:

Use Compute Optimizer to avoid future issues

Use AWS Compute Optimizer to get insights into your usage of Amazon EC2, Amazon ECS, and Lambda resources. The Compute Optimizer detects when your resource usage is high to help you avoid timeouts. It also provides best practices about how to optimize your workloads.

Related information

How do I troubleshoot high latency on my Application Load Balancer in Elastic Load Balancing (ELB)?

How do I troubleshoot authentication issues in my Application Load Balancer?

How do I troubleshoot 504 errors returned when I use a Classic Load Balancer?

AWS OFFICIALUpdated 3 months ago