How do I troubleshoot API Gateway HTTP invoke errors?

Resolution

To troubleshoot 5xx errors for REST APIs that you configured to log resource and operation requests, use the AWSSupport-TroubleshootAPIGatewayHttpErrors runbook. 

Or, you can manually troubleshoot 5xx errors.

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Use the automated runbook to troubleshoot errors

Before the AWSSupport-TroubleshootAPIGatewayHttpErrors runbook analyzes Amazon CloudWatch Logs, the runbook validates the API, resource, operation, and stage.

Grant the required permissions

Before you run the runbook, make sure that your AWS Identity and Access Management (IAM) user or role has the correct permissions. Verify that your IAM user or role has permission to access the runbook. 

Also, the IAM user or assumed service role that runs the automation must have the following permissions:

• apigateway:GET
• logs:GetQueryResults
• logs:StartQuery
• ssm:DescribeAutomationExecutions
• ssm:GetAutomationExecution
• ssm:DescribeAutomationStepExecutions
• ssm:StartAutomationExecution
• ssm:DescribeDocument
• ssm:GetDocument
• ssm:ListDocuments

Run the automation

Complete the following steps:

1. Open the AWSSupport-TroubleshootAPIGatewayHttpErrors runbook.
2. Choose Execute automation.
   For input parameters, enter the following:
   RestApiId: The API ID for the API that you're troubleshooting.
   StageName: The name of the deployed stage.
   ResourcePath: The operation's resource path.
   HttpMethod: The method for the configured resource path.
   StartTime: The start date and time to query the CloudWatch logs. The format must be yyyy-MM-ddTHH:mm:ss and the time zone must be UTC.
   EndTime: The end date and time to stop the query of the CloudWatch logs. The format must be yyyy-MM-ddTHH:mm:ss and the time zone must be UTC.
   AccessLogs: The access logs that you want the runbook to analyze.
   ExecutionId: The execution ID for the request that experiences errors.
   AutomationAssumeRole: The ARN of the IAM role that allows Automation to perform the actions on your behalf. If you don't specify a role, then Automation uses the permissions of the user that starts the runbook.
   StageName: The name of the deployed stage.
3. Choose Execute.
4. Review the Outputs section for detailed results.
   Note: The automation completes successfully, even when the runbook doesn't find logs.

Manually troubleshoot 5xx status code errors

Note: The following resolution applies only to REST APIs.

Before you begin, turn on Amazon CloudWatch Logs for troubleshooting API Gateway errors. To find 5xx errors in your run logs, see How do I find API Gateway REST API errors in my CloudWatch logs? The API Gateway 5XXError metric captures the number of server-side errors in a specified period.

Errors in the Lambda function code

For API endpoint 500 errors that integrate with AWS Lambda, see Error handling patterns in Amazon API Gateway and AWS Lambda.

Missing permissions for a stage variable

If you use a stage variable to set up an API Gateway to invoke a Lambda function, then you might receive an Internal server error. To resolve this issue, see I defined my Lambda integration in API Gateway using a stage variable. Why do I get an "Internal server error" and a 500 status code when I invoke the API method?

Incorrect or missing HTTP status code mapping

To resolve this issue, set up mock integrations in API Gateway.

Throttling issues

If a high number of requests throttles the backend service, then the API Gateway API might return an Internal server error. To resolve this issue, activate an exponential backoff and retry mechanism, and then try the request again. If you still experience the issue, then check your API Gateway quota. If you exceed the service quota, then you can request a service quota increase.

Undefined HTTP method of POST

For Lambda integration, you must use the HTTP method of POST for the integration request. Run the following put-integration AWS CLI command to update the method integration request:

aws apigateway put-integration \
    --rest-api-id id \ 
   --resource-id id \
    --http-method ANY \
    --type AWS_PROXY \
    --integration-http-method POST \
    --uri arn:aws:apigateway:us-east-2:lambda:path//2015-03-31/functions/arn:aws:lambda:us-east-2:account_id:function:helloworld/invocations

Note: Replace the rest-api-id with your REST API ID, resource-id with your resource ID, and the example URI with your URI.

Then, run the following create-deployment AWS CLI command to deploy the REST API:

aws apigateway create-deployment \
    --rest-api-id id \
    --stage-name stage-resource

Note: Replace rest-api-id with your REST API ID and stage-resource with your Stage resource for the Deployment resource to create.

Lambda permissions

Make sure that the resource-based policy of your Lambda function or Lambda authorizer includes permissions for your API to invoke the function. 

Lambda function output JSON format issue

If the output from your integrated Lambda function doesn't conform to the specified JSON format for REST APIs, then you receive an error. Make sure to use the correct JSON format for the output from Lambda functions for proxy integrations and from Lambda authorizers. Use the following example JSON formats.

Example Lambda function for proxy integration:

{
    "isBase64Encoded": true|false,
    "statusCode": httpStatusCode,
    "headers": { "headerName": "headerValue", ... },
    "multiValueHeaders": { "headerName": ["headerValue", "headerValue2", ...], ... },
    "body": "..."
}

Example Lambda authorizer:

{
   "principalId": "user",
  "policyDocument": {
    "Version": "2012-10-17",
    "Statement": [
      { "Action": "execute-api:Invoke",
        "Effect": "Deny",
        "Resource": "arn:aws:execute-api:us-west-2:123456789012:ymy8tbxw7b/dev/GET/"
      }
    ]
  }
}

Backend payload size exceeds 10 MB

The maximum HTTP API quota for a backend payload size is 10 MB. You can't increase the size, so make sure that the backend payload size doesn't exceed the 10 MB quota.

Private endpoint integration

If you use a private API endpoint, then you must also configure the API Gateway private integration.

Internal service failures

If AWS experiences internal service issues, then you might receive a 500 error. Wait for the issue to resolve within AWS or the API Gateway service, and then retry the request with exponential backoff.

502 status code error: Bad gateway

You receive A 502 status code Bad gateway when API Gateway can't process the response as a gateway or proxy. To troubleshoot this issue, see How do I resolve HTTP 502 errors from API Gateway REST APIs with Lambda proxy integration?

Note: When API Gateway reads the response from the backend service, it uses mapping templates to map the format in the integration response section. For more information, see Set up an integration response in API Gateway.

503 status code error: Service unavailable

A 503 status code error occurs because the backend integration isn't available, so the API Gateway API doesn't receive a response.

This error might occur for the following reasons:

• The backend server exceeded capacity and can't process new client requests.
• The backend server is completing temporary maintenance.

To resolve this issue, you can add more resources to the backend server and activate an exponential backoff and retry mechanism on the client. Then, try the request again.

504 status code error: Endpoint request timed out

If an integration request takes longer than your API Gateway REST API maximum integration timeout parameter, then API Gateway returns an HTTP 504 status code. To resolve this issue, see How can I troubleshoot API HTTP 504 timeout errors with API Gateway?

Related information

AWS Support Automation Workflows (SAW)

Security best practices in Amazon API Gateway

Monitoring REST API execution with Amazon CloudWatch metrics

How do I turn on CloudWatch Logs to troubleshoot my API Gateway REST API or WebSocket API?