Lambda throttle without reaching concurrency limit

I did not set the reserved concurrency but did set the provisioned concurrency, which is also not sure why the cloudwatch metric shows that provisioned concurrency execution is lower than the set value while the concurrency execution exceeded it.

It seems that you are hitting invocation requests per second quota, which means even if concurrency execution does not reach 1,000(over all account level concurrency), if the invocations are over 10,000 per second it will throttle.

I believe you'd have already gone through this to have clearer understanding, just for your reference:

   Concurrent execution is the total number of concurrent execution available in that account.
   Reserved concurrency is the maximum number of concurrent instances you want to allocate to your function.
   Provisioned concurrency is the number of pre-initialized execution environments you want to allocate to your function.

It seems that you are hitting invocation requests per second quota, which means even if concurrency execution does not reach 1,000(over all account level concurrency), if the invocations are over 10,000 per second it will throttle.

Go to Cloudwatch -> Metrics -> All Metric -> Source and paste the following query to see total number of invocations per second while doing the load test. This would explain the throttling in your case:

        {
            "metrics": [
                [ { "expression": "SEARCH('{AWS/Lambda,FunctionName} (\"Invocations\")', 'Sum', 300)", "label": "LI", "id": "e1", "region": "us-east-1" } ]
            ],
            "view": "timeSeries",
            "stacked": false,
            "region": "us-east-1",
            "stat": "Sum",
            "period": 300,
            "title": "Lambda Invocations"
        }

I believe you'd have already gone through this to have clearer understanding, mentioning here just for your reference from AWS Lambda documentation Lambda Concurrency:

1. Concurrent execution is the total number of concurrent execution available in that account. 2. Reserved concurrency is the maximum number of concurrent instances you want to allocate to your function. 3. Provisioned concurrency is the number of pre-initialized execution environments you want to allocate to your function.

thanks regarding my problem it is clear now that I hit the invocation request limit. I still don't understand the cloudwatch metric part tho.

I have cloudwatch metrics showing Concurrent Execution and Provisioned Concurrent Execution, both use MAX as statistic. I set the provisioned concurrency to 500, but most time the metrics show 500 > concurrent execution > provisioned concurrency.

My initial thought was is suppose to be using provisioned concurrency instead of using "Spill over provisioned concurrency". Can you help me understand this part? thanks

Like I mentioned above, provisioned concurrency is the number of pre-initialized execution environments that you want to allocate to your function to reduce latency and cold start. Important point to note here is, AWS Documentations says You can configure Provisioned Concurrency up to the Unreserved account concurrency in your account, minus 100. The remaining 100 units of concurrency are for functions that aren't using reserved concurrency. For example, if your account has a concurrency limit of 1,000, and you haven't assigned any reserved or provisioned concurrency to any of your other functions, you can configure a maximum of 900 provisioned concurrency units for a single function.

In your case, unreserved account concurrency is 400, so max 300 provisioned execution environments can be allotted, rest would be running using account level unreserved concurrency and as that unreserved concurrency limit would be exhausted, your lambda function would start getting throttled. Hope this helps. I'd highly encourage you to go through the concurrency documentation which I shared earlier, that would help you in a big way for better understanding.

Feel free to comment here if you have any additional questions, happy to help. If this answers your question, please Accept answer and upvote it. TIA.

I do not set reserve concurrency.

I did try to hit it from my local machine and it doesn't get throttled and can hit close to 1000 concurrent limit. yes I am load testing the function.

Regarding sum of invocation, does it means even tho the concurrency execution does not reach 1,000, if the invocations are over 10,000 per second it will throttle?

Another question is the cloudwatch metric shows that "concurrent execution" is higher than "provisioned concurrent execution" which is fine. but why the "provisioned concurrent execution" is below what is set?

I set the provisioned concurrency to 500, and during test it shows the provisioned concurrent execution 100~300 while at the same time the concurrent execution is more than 500.

To your questions: 1. Yes, if the invocations are over 10,000 per second, you will be throttled. and 2. There is another limit which is the RPS for provisioned concurrency functions. which 10 * the number set for provisioned concurrency for the function. If you set your PC to 500, you will be able to invoke the provisioned function 5000 RPS. The rest will go to regular functions.

BTW, to overcome the throttling issue without PC, just ask for a concurrency limit increase. For PC functions, you will need to configure higher number, which has higher cost. Make sure that you need PC.