Why isn't my Lambda function with an Amazon SQS event source optimally scaling?

Resolution

Note: When you configure an Amazon SQS queue as an event source, Lambda functions can optimally scale up to 300 more instances a minute. The maximum number of concurrent executions for standard queues is 1,250. If you use first-in, first-out (FIFO) event source mapping, then functions can scale in concurrency to the number of active message groups.

Identify and resolve Lambda function invocation issues

To prevent errors at scale, Lambda throttles function scaling when invocation issues occur. After you resolve the issues, Lambda continues to scale the function. For more information, see Backoff strategy for failed invocations. To resolve Lambda function invocation errors, see Handling errors for an SQS event source in Lambda and How do I troubleshoot Lambda function failures?

Configure your Lambda function with optimal concurrency

Reserved concurrency

If you configured reserved concurrency on your function, then Lambda throttles your function when the function reaches the reserved value. Event source mappings don't include reserved concurrency and can process more messages from your queue than send to your function.

The minimum reserved concurrency for your function must be 1,250 for standard SQS queues and equal to the total active message groups for FIFO queues. It's a best practice to set your reserved concurrency higher than the number of message groups in the FIFO queue. A lower reserved concurrency can cause processing delays from the FIFO queue and throttle your function.

Important: To limit the number of concurrent invocations, use the maximum concurrency setting for Amazon SQS event sources instead of reserved concurrency.

Default concurrency

Your AWS account has a default concurrency quota of 1,000 for all functions in the same account and AWS Region. Functions scale until they reach the maximum available concurrency. When all your functions use the pool of 1,000 concurrency, Lambda throttles invocations.

Lambda restricts how fast your functions can scale so that sudden spikes in traffic don't cause the functions to excessively scale. Lambda functions initially scale based on the concurrency scaling rate. Because the concurrency scaling rate is at the function level, each function in your account can scale independently of other functions. The concurrency scaling rate also differs from the account-level concurrency quota that provides the total amount of concurrency available to your functions.

Note: Default concurrency is different from provisioned concurrency. For more information, see Understanding reserved concurrency and provisioned concurrency.

Maximum concurrency setting

When you set the maximum concurrency for one event source, the value applies only to the specific event source. Additional event sources without maximum concurrency use the remaining account's concurrency quota or the reserved concurrency.

You can use the maximum concurrency setting and reserved concurrency together. It's a best practice to set the maximum concurrency lower than the function's reserved concurrency so that functions don't throttle.

Confirm that there are enough messages in your Amazon SQS queue to allow your Lambda function to scale

If you configured an Amazon SQS queue to invoke a Lambda function, then Lambda scales invocations only when there are messages in the queue.

To use Amazon CloudWatch to check how many messages in your queue must be processed, review the queue's ApproximateNumberOfMessagesVisible metric.

If the metric is low or at 0, then your function can't scale.

If the metric is high and there aren't invocation issues, then increase the batch size on your event notification until the Duration metric doesn't spike. For more information, see Types of metrics for Lambda functions. 

Note: The maximum batch size for a standard Amazon SQS queue is 10,000 records. For FIFO queues, the maximum batch size is 10 records.

Confirm that Amazon SQS added the messages to a queue

To verify that Amazon SQS added the new messages to a queue, review the queue's NumberOfMessagesSent metric.

Confirm that the consumer made the ReceiveMessage API call

To verify that the consumer made the ReceiveMessage API call, review the queue's NumberOfMessagesReceived metric.

Note: Spikes in the NumberOfEmptyReceives metric show that the consumer made the ReceiveMessage API call, but the call doesn't return a message from the queue. This issue occurs when no messages are in the queue.

Use provisioned mode with Amazon SQS event source mappings

Use provisioned mode for workloads to fine tune the throughput of your event source mapping and define the limits for the event pollers. Amazon SQS event source mapping configured with provisioned mode scales 3x faster and 16x higher concurrency than the default Amazon SQS event source mapping capability.

It's a best practice to use provisioned mode for SQS event driven workloads that have strict performance requirements. For example, financial services firms that process market data feeds and e-commerce platforms that provide real-time personalized recommendations.

For more information, see AWS Lambda enhances event processing with provisioned mode for SQS event-source mapping.

Note: Provisioned mode incurs additional costs. For more information, see AWS Lambda pricing.

Related information

Using Lambda with Amazon SQS

Managing AWS Lambda function concurrency