Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

Resolve "Endpoint request timed out" 504 errors when API Gateway invokes a Lambda function for long-running operations

6 minute read
Content level: Intermediate
1

I want to resolve the "Endpoint request timed out" error with HTTP status code 504 that I receive when my Lambda function calls an Amazon API Gateway endpoint that triggers a downstream Lambda performing long-running operations.

Short description

When you call an Amazon API Gateway REST API endpoint, the API Gateway integration enforces a default timeout of 29 seconds. If the integration backend (such as an AWS Lambda function) does not respond within this window, API Gateway terminates the connection and returns the following error to the caller:

"error": "AutomationFailure(message={'message': 'Endpoint request timed out'}, statusCode=504)" "Error Response Body: {"message": "Endpoint request timed out"}" This commonly happens when the downstream Lambda processes a workload that grows in size over time, such as iterating through 30 or more servers, making multiple AWS API calls per server. With small workloads the call completes within 29 seconds, but larger workloads exceed the limit and fail.

Resolution

The 29-second default integration timeout can be increased up to 90 seconds (and higher with a support case) for Regional and private REST APIs. Resolution requires changes in four places: the account-level service quota, the specific API method, the API stage deployment, and the Lambda function timeouts.

Note: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, see Troubleshooting errors for the AWS CLI. Also make sure you are using the most recent AWS CLI version.

Increase the API Gateway integration timeout quota

To raise the integration timeout quota at the account level, complete the following steps:

  1. Open the AWS Service Quotas console.
  2. In the navigation pane, choose AWS services.
  3. In the search field, enter Amazon API Gateway, and then choose the service.
  4. From the quota list, select Maximum integration timeout in milliseconds.
  5. Choose Request increase at account level.
  6. For Increase quota value, enter the desired value in milliseconds. For example, enter 90000 for 90 seconds.
  7. Choose Request.

The status changes to Pending and then to Approved, usually within a few minutes for values within the maximum allowed quota. To request a value above 90,000 ms, submit a support case with your business justification.

Note: Quota increases for the integration timeout apply only to Regional and private REST APIs. Edge-optimized REST APIs and HTTP APIs are not eligible.

Update the integration timeout on your API method

The quota increase by itself does not change the timeout for any existing API. Each method's integration timeout must be updated explicitly. To update the timeout on your method, complete the following steps:

  1. Open the API Gateway console.
  2. In the navigation pane, choose APIs, and then select your REST API.
  3. In Resources, select the method (such as POST /mgn).
  4. Choose the Integration request tab.
  5. Choose Edit for Integration request settings.
  6. For Integration timeout, enter the new value in milliseconds, such as 90000.
  7. Choose Save.

Redeploy the API stage

Changes to the integration request settings do not take effect until the API is redeployed.

  1. From the API Gateway console with your API selected, choose Deploy API from the Actions menu.
  2. Select your stage (such as prod).
  3. Add a deployment description if desired.
  4. Choose Deploy.

Important: Skipping the redeploy step is the most common reason the new timeout does not apply. The console saves the configuration but the live endpoint continues to use the previous value until the stage is redeployed.

Update the Lambda function timeouts

The two Lambda functions involved in this workflow each have their own timeout setting. Both must be greater than the API Gateway integration timeout, plus a buffer.

Calling Lambda function

The Lambda that makes the HTTP call to API Gateway must wait for API Gateway to respond. Set its timeout higher than the integration timeout.

  1. Open the Lambda console.
  2. Choose Functions, and then select the calling Lambda function.
  3. Choose the Configuration tab, and then choose General configuration.
  4. Choose Edit.
  5. For Timeout, enter a value greater than the API Gateway integration timeout. For a 90-second integration timeout, set Lambda timeout to at least 2 minutes.
  6. For Memory, allocate at least 1024 MB to provide adequate CPU performance.
  7. Choose Save.

Downstream Lambda function

The Lambda invoked by API Gateway must complete its work within its own timeout. For long-running workloads, configure the function with adequate timeout and memory.

  1. From the Lambda console, select the downstream Lambda function.
  2. Choose Configuration > General configuration > Edit.
  3. For Timeout, enter a value greater than or equal to the API Gateway integration timeout. A value of 5 minutes provides headroom for variability.
  4. For Memory, set a value appropriate for the workload. For workloads making many parallel AWS API calls (for example, 30 or more EC2 or MGN describe operations), 2048 MB is a good starting point.
  5. Choose Save.

Important: A Lambda timeout that is too low (such as the default 3 seconds for newly created functions) silently terminates the function, leading to misleading error patterns. Always confirm the Lambda timeout exceeds the longest expected synchronous downstream call.

Verify the configuration

After applying all changes, trigger a fresh invocation and verify the workflow:

  1. Open the CloudWatch console.
  2. In the navigation pane, choose Logs > Log groups.
  3. Select the log group for the calling Lambda. Confirm the function runs to completion without 504 errors.
  4. Select the log group for the downstream Lambda. Confirm new invocations appear that align with the expected timing.
  5. In the REPORT line for any recent invocation, check the Duration field. The value should be less than the configured Lambda timeout.

To view the integration timeout currently applied to your method, run the following AWS CLI command:

aws apigateway get-integration --rest-api-id YourApiId --resource-id YourResourceId --http-method POST --query 'timeoutInMillis'

Replace YourApiId with your REST API ID and YourResourceId with the resource ID of the method.

Related information

No comments

Relevant content