How do I troubleshoot Lambda function invocation timeout errors?

Resolution

Lambda functions can time out for a variety of reasons. To troubleshoot Lambda function timeouts, first determine what's causing the issue by using one of the AWS services and features listed in this article. Then, remediate the problem based on your use case.

To help prevent your Lambda function from timing out, see the Best practices for preventing Lambda function timeouts section of this article.

Verify that your Lambda function is timing out

Retrieve the request IDs of any timed-out invocations by searching the function's Amazon CloudWatch log group for the phrase Task timed out. Then, use the request IDs of the associated timed-out invocations to retrieve the full logs for each invocation timeout.

For instructions, see How do I determine if my Lambda function is timing out?

Identify what's causing your Lambda function to time out

Use one or more of the following methods to identify the point of failure that's causing your function to time out:

Review your CloudWatch Logs for Lambda

You can use Amazon CloudWatch to view all logs generated by your function's code and identify potential issues. For instructions, see Accessing CloudWatch Logs for Lambda.

If your function is returning a stack trace, then the error message in the stack trace specifies what's causing the error.

Important: Lambda generates three log lines for each invocation automatically (START, END, and REPORT). These three lines are the only lines that appear in your function's CloudWatch logs if either of the following is true:

• There's no other explicit logging configured in the Lambda function's custom code.
• The function's duration limit is reached before Lambda can run the function's code that outputs logs.

If you can't determine what's causing the timeouts after reviewing the logs, try one or more of the following solutions:

• Verify that the retry count and timeout settings on the AWS SDK that you're using allow enough time for your function to initialize.
• Increase the Lambda function's timeout setting temporarily to allow enough time for the function code to generate log data.
• Increase the function's configured memory to help reduce invocation duration latency and increase computing power.

To add more logging output to your function's code, see the following documentation for the Lambda runtime that you're using:

• AWS Lambda function logging in Node.js
• AWS Lambda function logging in Python
• AWS Lambda function logging in Ruby
• AWS Lambda function logging in Java
• AWS Lambda function logging in Go
• AWS Lambda function logging in C#
• AWS Lambda function logging in PowerShell

Use AWS X-Ray to identify any code performance bottlenecks

If your Lambda function uses downstream AWS resources, microservices, databases, or HTTP web APIs, then you can use AWS X-Ray to help troubleshoot code performance issues.

For more information, see Using AWS Lambda with AWS X-Ray.

Use Lambda Insights to collect system-level metrics for your function

Lambda Insights collects system-level metrics, including CPU time, memory, disk, and network metrics. It also collects diagnostic information, including cold starts and Lambda worker shutdowns to help you isolate issues with your Lambda functions.

For more information, see Using Lambda Insights.

Note: Using Lambda Insights incurs charges to your AWS account. You're charged for the invocation time consumed by the Lambda extension in 1 ms increments.

Use VPC Flow Logs to determine why a specific invocation request was denied or didn't route

VPC Flow Logs allow you to see all the network traffic flowing to and from an Amazon Virtual Private Cloud (Amazon VPC).

For more information, see Troubleshoot networking issues in Lambda.

Note: Keep in mind the following if you choose to set up VPC Flow Logs:

• Data ingestion and archival charges for vended logs apply when you publish flow logs to either of the following:

• CloudWatch Logs
• Amazon Simple Storage Service (Amazon S3)

• When you configure a Lambda function to access resources in an Amazon VPC, Lambda assigns the function to an elastic network interface. To identify the network traffic associated with your Lambda function, you must locate the function's network interface. For instructions, see Why can't I detach or delete an elastic network interface that Lambda created?

Use HTTP wire traces for detailed logging of network requests that are generated by the function's code during an invocation

For more information, see Logging HTTP wire traces.

Best practices for preventing Lambda function timeouts

Make sure that your Lambda function is idempotent

API calls can take longer than expected due to transient network issues. Network issues can also cause retries and duplicated API requests. To prepare for these occurrences, make sure that your Lambda function is idempotent.

For more information, see How do I make my Lambda function idempotent?

Initialize your function's static logic outside of the function handler

When you initialize a Lambda function, Lambda allocates up to 10 seconds for the Init phase of the invocation to complete. Because of this time constraint, it's a best practice to do the following outside of the function handler in initialization code:

• Import libraries and dependencies
• Set up initial configuration
• Initialize connections to other downstream services and resources

This static initialization allows these resources to be initialized once per sandbox and then reused for all future invocations in the same runtime environment.

For more information, see Optimizing static initialization. Also, Downstream unavailability in the Lambda Operator Guide and Function code in the Lambda Developer Guide.

Note: Lambda removes idle connections to downstream resources. To allow your function to maintain a persistent connection, use the tcp_keepalive variable that's associated with the Lambda runtime that you're using.

Verify that the retry count and timeout settings on the AWS SDK that you're using allow enough time for your function to initialize

If you make an API call using an AWS SDK and the call fails, then the AWS SDK automatically retries the call. How many times the AWS SDK retries and for how long is determined by settings that vary between each AWS SDK. Your function might need more time to initialize than the default AWS SDK settings allow.

For more information, see How do I troubleshoot retry and timeout issues when invoking a Lambda function using an AWS SDK?

(Optional) Configure provisioned concurrency for your Lambda function

Provisioned concurrency initializes a requested number of runtime environments so that they're prepared to respond immediately to your function's invocations. To set up provisioned concurrency for your function, follow the instructions in Configuring provisioned concurrency.

Note: Configuring provisioned concurrency incurs charges to your AWS account. You can configure provisioned concurrency on a version of a function or on a Lambda function alias.

Verify that your Lambda function has enough system resources

The amount of network bandwidth and CPU allocated to a Lambda function invocation is determined by the function's memory configuration.

For more information, see Memory and computing power in the Lambda Operator Guide.

Make sure that any background processes your Lambda function uses complete before the function handler returns a string

For more information, see Understanding container reuse in AWS Lambda.

Verify that your Lambda function is configured to work within the maximum timeout settings of any integrated AWS services

Even though a Lambda function's maximum invocation timeout limit is 15 minutes, other AWS services may have different timeout limits.

For example, Amazon API Gateway waits a maximum of 29 seconds for a Lambda function proxy invocation to complete. For more information, see How can I resolve the errors that I receive when I integrate API Gateway with a Lambda function? Also, see Using AWS Lambda with other services.

Confirm that there's a valid network path to the endpoint that your function is trying to reach

To review your network settings, follow the instructions in How do I troubleshoot timeout issues with a Lambda function that's in an Amazon VPC?

---