How do I make my Lambda function idempotent?

3 minute read
2

I want to make my AWS Lambda function code idempotent to prevent inconsistencies and data loss in my application.

Short description

To make your AWS Lambda function idempotent, you must design your function logic to correctly handle duplicated events. The function's code must properly validate input events and identify whether events were previously processed. Your application's functionality determines the best way to write the code.

Idempotent function logic can help reduce the following issues:

  • Unnecessary API calls
  • Code processing time
  • Data inconsistency
  • Throttles
  • Latency

Resolution

The following example function logic and best practices apply to most uses.

Idempotent Lambda function logic example

To use the idempotent Lambda function, complete the following steps:

  1. Extract the value of a unique attribute of the input event, such as a transaction or purchase ID.
  2. Use a conditional expression to put the idempotent key into your Amazon DynamoDB table.
  3. For the record attributes, take the following actions:
    Set Status to IN_PROGRESS.
    For in_progress_expiry timestamp, choose current timestamp in milliseconds + the number of milliseconds left before the execution times out. For python, use context.get_remaining_time_in_millis.
    Note: With in_progress_expiry, the Lambda function can safely retry the logic.
    If the PutItem from the previous step returns ConditionalCheckFailedException, then the idempotent key is processing.
    To use the DynamoDB Time To Live (TTL) attribute with idempotent_expiry, set the attribute to a later time for the value.
  4. Update the key record status to COMPLETE.
  5. Complete the action.

Note: When you add AWS services to your architecture, you might incur additional costs. For more information, see Amazon DynamoDB pricing and AWS Pricing.

Lambda function idempotency best practices

Use the following best practices:

  • Plan your idempotency features before you develop your application.
  • When your Lambda code processes a duplicate event, make sure that the process ends without an error.
    Note: A process that ends in error can cause services that invoke your function to further retry.
  • Configure the Lambda function timeout setting so that the full runtime is correctly processed.
    Note: If you use a separate service to persist data and control duplicated events, then you might need to make API calls to HTTPS endpoints. API calls to HTTPS endpoints might require more runtime than the default 3 seconds.
  • Test and optimize your function as much as possible. Simulate a real scenario and rate of requests.
    Note: It's critical to test and optimize idempotent function logic to help prevent potential timeouts, excessive latency, or bottlenecks.
  • To store session data, use a service that's easily scalable and provides high throughput, such as DynamoDB.

Note: When you make API calls to Amazon Elastic Compute Cloud (Amazon EC2), you can use the clientToken parameter. This parameter makes sure that a mutating API request successfully completes its workflow only once, even when you initiate multiple retries with the same clientToken.

Related information

Understand the Lambda programming model

Create a Lambda function with the console

Making retries safe with idempotent APIs

AWS OFFICIAL
AWS OFFICIALUpdated 5 months ago
2 Comments

You can use Powertools for AWS Lambda, an open source library, to add idempotency to your Lambda functions. Powertools for AWS Lambda is a developer toolkit to implement Serverless best practices and increase developer velocity, and is available for Python, Java, TypeScript and .NET.

See more at https://github.com/aws-powertools

profile pictureAWS
replied 2 years ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
EXPERT
replied 2 years ago