By using AWS re:Post, you agree to the AWS re:Post Terms of Use

How to debug 504 timeouts from an application load balancer

0

We're experiencing intermittent 504 errors after ~1 minute from our API running in Fargate with an ALB in front of it. The time and 504 seems to indicate there was an idle timeout, according to this document: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-troubleshooting.html#http-504-issues. The vast majority of requests go through just fine, but something on the order of 1 in 5,000 will error. However, our API logs are indicating that the request was received, processed, and responded to. This is a sample of a 504 from the ALB's logs. "abcdef" where data redacted.

h2 2024-01-26T14:49:30.637331Z app/api-prod/abcdef 108.49.138.101:36790 10.0.0.210:4700 0.000 -1 -1 504 - 229576 604 "POST https://abcdef/participantTest/audio/score HTTP/2.0" "Mozilla/5.0 (X11; CrOS x86_64 14541.0.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:us-east-2:abcdef:targetgroup/api-prod/abcdef "Root=1-65b3c63e" "abcdef.com" "session-reused" 0 2024-01-26T14:48:30.597000Z "forward" "-" "-" "10.0.0.210:4700" "-" "-" "-"

The POST requests are usually around 1-20MBs, and so we expect the request to take a little while for clients to upload, but no where near 1 minute, and usually our endpoints indicate the processing time in our API takes on the order of 500ms to complete.

Are there any common causes for something like this, given how randomly it seems to occur? Is there another way to get more information about the traffic between Fargate and the ALB? What are some other debugging steps we could take?

asked 10 months ago462 views
1 Answer
0

Hello.

You may find the troubleshooting methods in the following documentation helpful.
https://repost.aws/knowledge-center/504-error-alb

profile picture
EXPERT
answered 10 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions