How do I troubleshoot HTTP 504 errors from an API Gateway REST API with a Lambda backend?

3 minute read

To troubleshoot HTTP 504 errors from a REST API using Amazon API Gateway with an AWS Lambda backend, first set up Amazon CloudWatch API logging and check CloudWatch Logs Insights.

Resolution

When your REST API requests return an HTTP 504 error status code, you must run the following checks.

Check for IntegrationLatency spikes

To check if there's an IntegrationLatency spike in API Gateway, review the integration latency duration time. To see the IntegrationLatency duration time, configure the access logging variable $context.integration.latency for HTTP API logging. For more information, see Set up CloudWatch API logging using the API Gateway console.

An IntegrationLatency spike in API Gateway indicates that the request spent most of its time in Lambda. To confirm this step, check the Lambda function's Duration performance metric. for more information, see Performance metrics.

Review requests with CloudWatch Logs Insights

Use Amazon CloudWatch Logs Insights to review requests that resulted in "504 errors." To review the requests, on the CloudWatch console, in the navigation pane, choose Logs, and then choose Log Insights. Select your API Gateway log group. Then, set the relative time with one of the following queries:

parse @message '(*) *' as reqId, message
| filter message like /Method completed with status: \d\d\d/
| parse message 'Method completed with status: *' as status
| filter status = 504
| sort @timestamp desc
| limit 20

-or-

fields @timestamp, @message
|filter message like /Method completed with status: 504/
| sort @timestamp desc
| limit 20

Instrument X-Ray traces

If "504 errors" continuously occur, then determine where the Lambda function is spending its time. Configure AWS X-Ray traces for Lambda functions based on the Lambda function runtime.

For Python:

from aws_xray_sdk.core import xray_recorder
from aws_xray_sdk.core import patch_all
patch_all()

For Node.js:

const AWSXRay = require('aws-xray-sdk-core')
const AWS = AWSXRay.captureAWS(require('aws-sdk'))

Note: After you configure AWS X-Ray traces on a Lambda function, you must build a new deployment package.

Implement API retries

When a "504 error" occurs and the request isn't found in Lambda, implement API retries on the client. The error can result from a temporary network failure in API Gateway. For more information, see Understanding retry behavior in Lambda,

Check the Lambda function configuration

Make sure that your Lambda function has only API Gateway event-specific processing logic. This logic helps to minimize the time it takes for the Lambda function to run and helps the function keep up with incoming events.

Note: API Gateway REST API's default maximum integration timeout is 29 seconds. You can increase the integration timeout to more than 29 seconds for AWS Regional APIs and private APIs. This increase might require a reduction in your AWS Account-level throttle quota limit.

For more information, see Set up asynchronous invocation of the backend Lambda function.