Why did my Lambda function retry valid Amazon SQS messages and place them in my dead-letter queue?

Short description

If your Lambda function is throttled, returns an error, or doesn't respond when reading an Amazon SQS message batch, then the messages return to your queue. After the visibility timeout occurs, your Lambda function receives the message batch again. If your function fails to process valid messages multiple times, then Amazon SQS sends the messages to your dead-letter queue, if you've configured one.

To prevent valid messages from being placed in a dead-letter queue, your function code must be idempotent and able to handle messages multiple times. For more information, see How can I prevent an Amazon SQS message from invoking my Lambda function more than once?

Resolution

Verify that your Lambda function's code is idempotent

For idempotency best practices and example function logic, see How do I prevent duplicate Lambda events?

Verify that your Amazon SQS queue's visibility timeout is at least six times longer than your Lambda function's timeout setting

Set your source queue's visibility timeout to at least six times longer than your function's timeout. The extra time allows your function to retry processing a batch if the function is throttled while processing a previous batch.

For more information, see Setting and adjusting the visibility timeout.

Note: If your function doesn't receive messages because the queue's visibility timeout isn't long enough, then the messages aren't recorded in your Amazon CloudWatch Logs.

Verify that the maxReceiveCount attribute is set to at least five on your source queue redrive policy

Set the maxReceiveCount on the source queue's redrive policy to at least five. If your function returns an error, or can't invoke because it's at maximum concurrency, processing might succeed with additional attempts. A maxReceiveCount of at least five gives your messages more chances to process before they're sent to your dead-letter queue.

To update the maximum concurrency setting limit, see Configuring maximum concurrency for Amazon SQS event sources.

For more information, see Using dead-letter queues in Amazon SQS.

Check the Lambda function for throttling and reserved concurrency

Lambda functions are sometimes throttled to protect your resources and downstream applications. Even though Lambda automatically scales to accommodate incoming traffic, your function can still throttle for various reasons.

Use the Lambda console to check the setting for reserved concurrency. If reserved concurrency isn't configured, then the function uses unreserved concurrency. When invocations with functions exceed the unreserved concurrency, throttling occurs.

Note: If you configured a function with zero reserved concurrency, then the function is throttled because it can't process any events. Make sure that you increase the value to a number greater than zero.

If other functions in the same AWS Region use the concurrency limit, then throttling can occur. Throttled Lambda functions send the SQS messages to the dead-letter queue.

Avoid reprocessing all SQS messages in a failed batch

All SQS messages are visible in the dead-letter queue for Lambda functions that get errors when processing a batch, including successful messages. Lambda then retries the entire batch of SQS messages again.

To avoid reprocessing all SQS messages in a failed batch, configure your event source mapping with the value ReportBatchItemFailures in the FunctionResponseTypes list. This event source mapping lets your function return a partial success and reduce the number of retries. For more information, see Reporting batch item failures for Lambda functions with an Amazon SQS trigger.

Identify and resolve any errors that your Lambda function returns

Follow the instructions in How do I troubleshoot Lambda function failures? Your function automatically removes messages from your queue only if the function doesn't return an error.

Related information

How do I request a concurrency limit increase for my Lambda function?

How do I troubleshoot Lambda function "Rate exceeded" and "TooManyRequestsException" throttling errors?