Tracing API Gateway 5xx errors

0

We have an API Gateway REST API (very standard, no auth, nothing interesting) which calls a bunch of Lambdas

Since yesterday ~12:00 GMT, we've been receiving a large number of 5xx errors, triggering one of our slack notifications. None of our lambdas have any errors (searched via metric filter and looking at the Cloudwatch metrics for lambdas), and the API GW execution logs are also clean. As far as we can see, all of our services are functioning normally, our monitoring services are all green, and we've had no client reports of outages/degraded services. There have been no deployments since last week, and there's no change in any other metric across the system. API GW only tells us that this is 5xx and not something more specific (such as a timeout).

How can we debug this? Ideally we'd like to see what the exact status/message is from these errors and where they are coming from. It could easily be that someone is hitting our API with large payloads or bad data, but we can't tell as it's not even getting to our lambdas.

已提問 1 個月前檢視次數 141 次
1 個回答
1
已接受的答案

To trace and debug 5xx errors in your API Gateway when Lambdas show no errors and logs are clean, follow these steps:

  • Enable access logging in API Gateway for detailed request/response logs.
  • Inspect CloudWatch access logs for patterns or specific requests causing 5xx errors.
  • Turn on AWS X-Ray for API Gateway and Lambda for end-to-end request tracing.
  • Analyze CloudWatch metrics for API Gateway, focusing on 5xx and backend connection errors.
  • Confirm no changes in API configuration that might affect integration responses or timeouts.
  • Check for requests with large payloads that could exceed size limits for API Gateway and Lambda.
profile picture
專家
已回答 1 個月前
profile picture
專家
已審閱 1 個月前
  • Thanks, this helped me find the source. Specifically, enabling the access logs and analyzing the IP addresses in question. Turns out we were under a distributed attack for exactly 24 hours.

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南