How do I troubleshoot 504 errors returned when I use a Classic Load Balancer?

3 minute read
0

I want to troubleshoot HTTP 504 errors I see in my Classic Load Balancer access logs or Amazon CloudWatch metrics. Or, I receive errors HTTP 504 errors when I connect to my service through a Classic Load Balancer.

Resolution

An HTTP 504 error means that a gateway or proxy has timed out. To troubleshoot, review the following possible causes:

Check your load balancer's idle timeout, and then modify if necessary

The most common reason for an HTTP 504 error is that a corresponding instance didn't respond to the request within the configured idle timeout. By default, the idle timeout for a Classic Load Balancer is 60 seconds.

Check CloudWatch metrics for your load balancer. When latency data points are equal to your configured load balancer timeout value, and HTTPCode_ELB_5XX has data points, at least one request has timed out.

To resolve this, complete one of the following actions:

  • Modify the idle timeout for your load balancer so that the HTTP request is completed within the idle timeout period.
  • Tune your application to respond faster.

Make sure that your backend instances keep connections open

When the backend instance closes a TCP connection to the load balancer before it reaches its idle timeout value, an HTTP 504 error might appear.

To resolve this issue, activate keep-alive settings on your backend instances. Then, set the keep-alive timeout to a value greater than the load balancer's idle timeout.

Turn off TCP_DEFER_ACCEPT (Apache)

When TCP_DEFER_ACCEPT is activated for Apache backend instances, the load balancer sends SYN to the backend instance to initiate a connection. Then, the backend instance responds with SYN-ACK, and the load balancer sends an empty ACK to the backend instance. Because the last ACK is empty, the backend doesn't accept the ACK, and instead resends a SYN-ACK to the load balancer. This can result in a subsequent SYN retry timeout.

When FIN or RST aren't sent before the backend instance closes the connection, the load balancer considers the connection established. Then, the load balancer sends requests through the TCP connection, and the backend responds with RST. This response generates a 504 error.

To resolve this issue, set both AcceptFilter http and AcceptFilter https to none.

Turn off the event MPM, and optimally configure the prefork and worker MPMs (Apache)

It's a best practice to turn off the event Multi-Processing Module (MPM) on backend instances that are registered to a load balancer. Because the Apache backend dynamically closes connections, these closed connections might result in HTTP 504 errors. For more information, see Apache MPM event on the Apache website.

For optimal performance when you use the prefork and worker MPMs, use the following values:

mod_prefork MPMmod_worker MPM
Timeout6565
KeepAliveTimeout6565
KeepAliveOnOn
MaxKeepAliveRequests100000
AcceptFilter httpnonenone
AcceptFilter httpsnonenone

Note: The values are based on a load balancer that is configured with a 60-second idle timeout.

For more information, see Apache MPM worker and Apache MPM prefork on the Apache website.

Related information

Monitor your Classic Load Balancer

Troubleshoot a Classic Load Balancer: HTTP errors

What are the optimal settings that I can use for Apache or NGINX as a backend server for Elastic Load Balancing (ELB)?

AWS OFFICIAL
AWS OFFICIALUpdated 2 months ago