I want to make my AWS Lambda function code idempotent to prevent inconsistencies and data loss in my application.
Short description
To make your AWS Lambda function idempotent, you must design your function logic to correctly handle duplicated events. The function's code must properly validate input events and identify whether events were previously processed. Your application's functionality determines the best way to write the code.
Idempotent function logic can help reduce the following issues:
- Unnecessary API calls
- Code processing time
- Data inconsistency
- Throttles
- Latency
Resolution
The following example function logic and best practices apply to most uses.
Idempotent Lambda function logic example
To use the idempotent Lambda function, complete the following steps:
- Extract the value of a unique attribute of the input event, such as a transaction or purchase ID.
- Use a conditional expression to put the idempotent key into your Amazon DynamoDB table.
- For the record attributes, take the following actions:
Set Status to IN_PROGRESS.
For in_progress_expiry timestamp, choose current timestamp in milliseconds + the number of milliseconds left before the execution times out. For python, use context.get_remaining_time_in_millis.
Note: With in_progress_expiry, the Lambda function can safely retry the logic.
If the PutItem from the previous step returns ConditionalCheckFailedException, then the idempotent key is processing.
To use the DynamoDB Time To Live (TTL) attribute with idempotent_expiry, set the attribute to a later time for the value.
- Update the key record status to COMPLETE.
- Complete the action.
Note: When you add AWS services to your architecture, you might incur additional costs. For more information, see Amazon DynamoDB pricing and AWS Pricing.
Lambda function idempotency best practices
Use the following best practices:
- Plan your idempotency features before you develop your application.
- When your Lambda code processes a duplicate event, make sure that the process ends without an error.
Note: A process that ends in error can cause services that invoke your function to further retry.
- Configure the Lambda function timeout setting so that the full runtime is correctly processed.
Note: If you use a separate service to persist data and control duplicated events, then you might need to make API calls to HTTPS endpoints. API calls to HTTPS endpoints might require more runtime than the default 3 seconds.
- Test and optimize your function as much as possible. Simulate a real scenario and rate of requests.
Note: It's critical to test and optimize idempotent function logic to help prevent potential timeouts, excessive latency, or bottlenecks.
- To store session data, use a service that's easily scalable and provides high throughput, such as DynamoDB.
Note: When you make API calls to Amazon Elastic Compute Cloud (Amazon EC2), you can use the clientToken parameter. This parameter makes sure that a mutating API request successfully completes its workflow only once, even when you initiate multiple retries with the same clientToken.
Related information
Understand the Lambda programming model
Create a Lambda function with the console
Making retries safe with idempotent APIs