Understanding reading rate limit from a single prefix in S3

0

The published limits state that one can perform 5,500 GET requests per second on a single prefix of S3 bucket. I have a Lambda function that reads 400 objects of 38.4KB each, all in the same prefix, in under a second; I'll call this function L1. I have another Lambda function that invokes L1 in parallel 100 times, to a total of 40,000 GET requests. During some runs, this latter function is done under 1.4 seconds, which means some 28,500 GET requests per second. However, sometimes it times out, which means over 10 seconds in my configuration. Why is it that sometimes I can go over the limit? Why does my Lambda function time out sometimes (it should be able to read 40,000 objects under 10 seconds)?

asked 6 months ago198 views
1 Answer
0

Hello!

Appreciate the thorough explanation for your concern, to start, there are a few reasons why you might be exceeding the GET requests limit, will list them below.

  1. The 5,500 request per second limit is a guideline, not a hard limit. It is possible to burst over that limit for short periods of time before throttling kicks in. So when you execute a large parallel read like that, you can exceed 5,500 temporarily.
  2. The timeouts you see can be caused by eventual throttling or errors when going over the limits. Even if you don't hit the limits initially, a burst of requests like that may have a cascading effect that leads to throttles later on as S3 struggles to keep up.
  3. Resource constraints in your Lambda function like CPU, memory, network bandwidth or even just suboptimal code can cause timeouts even when S3 performs well.

Summary: Published S3 limits are not absolute guarantees, and exceeding them temporarily is possible, but leads to unreliable performance and throttling issues over time. I would optimize the parallel execution design to stay well below the documented limits for stable throughput. Things like Lambda exponential backoff and caching can help handle spikes when they do happen, but we want to try and stay below these limits.

answered 4 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions