Tracing API Gateway 5xx errors

0

We have an API Gateway REST API (very standard, no auth, nothing interesting) which calls a bunch of Lambdas

Since yesterday ~12:00 GMT, we've been receiving a large number of 5xx errors, triggering one of our slack notifications. None of our lambdas have any errors (searched via metric filter and looking at the Cloudwatch metrics for lambdas), and the API GW execution logs are also clean. As far as we can see, all of our services are functioning normally, our monitoring services are all green, and we've had no client reports of outages/degraded services. There have been no deployments since last week, and there's no change in any other metric across the system. API GW only tells us that this is 5xx and not something more specific (such as a timeout).

How can we debug this? Ideally we'd like to see what the exact status/message is from these errors and where they are coming from. It could easily be that someone is hitting our API with large payloads or bad data, but we can't tell as it's not even getting to our lambdas.

1개 답변
1
수락된 답변

To trace and debug 5xx errors in your API Gateway when Lambdas show no errors and logs are clean, follow these steps:

  • Enable access logging in API Gateway for detailed request/response logs.
  • Inspect CloudWatch access logs for patterns or specific requests causing 5xx errors.
  • Turn on AWS X-Ray for API Gateway and Lambda for end-to-end request tracing.
  • Analyze CloudWatch metrics for API Gateway, focusing on 5xx and backend connection errors.
  • Confirm no changes in API configuration that might affect integration responses or timeouts.
  • Check for requests with large payloads that could exceed size limits for API Gateway and Lambda.
profile picture
전문가
답변함 한 달 전
profile picture
전문가
검토됨 한 달 전
  • Thanks, this helped me find the source. Specifically, enabling the access logs and analyzing the IP addresses in question. Turns out we were under a distributed attack for exactly 24 hours.

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인