How can I schedule a lambda to be called with an array of events?

Question

Hi!
I'm fetching Odata entities and storing them to S3 with a fairly simple lambda. The source of Odata has 400k entries and fetching and storing 2k takes about 2min each, so I would expect a total of 200 minutes running time. Of source, the lambda times out after around 14k entries fetched and stored to S3.

I've implemented paging with $top and $skip in the function - is there a way I can schedule a chain of events to be sent to the lambda without creating 10-20 different EventBridge schedules with different event payloads?

Answer

Use Step Functions that call the Lambda function in a loop until the function returns that there are no more entries. If you know the params for invoking the functions up front, you can also use a Map state to iterate over these values and then you can run all of them in parallel.

How can I schedule a lambda to be called with an array of events?

Relevant content