How to use AWS services to run a function at specific scheduled times based on a provided list of date/time for an automation system?

Question

I am building an automation system where I need to run a function (e.g., perform data scraping, send notifications) at specific scheduled times. These scheduled times will be provided as a list of date/time strings in the format like "YYYY-MM-DD/HH.MM".

For example, if the list contains `["2024-03-12/10.30", "2024-03-12/12.30", "2024-03-13/10.30", "2024-04-12/17.30"]`, I need to run the function at the following times:
- 2024-03-12 10:30 AM
- 2024-03-12 12:30 PM
- 2024-03-13 10:30 AM
- 2024-04-12 5:30 PM

**Requirements**
- The list of scheduled times can be dynamic and may change regularly, so the solution should be able to handle updates to the list without significant effort.
- The solution should be scalable and able to handle a large number of scheduled events (potentially hundreds).

**Challenges:**
- Parsing and processing the provided list of date/time strings to create scheduled events.
- Integrating the function with the AWS service(s) responsible for triggering the function at the scheduled times.

I am familiar with AWS Lambda, but I'm not sure if it can be triggered with many and dynamic tasks, EventBridge's cron job scheduling seems limited for this use case. I'm looking for guidance on the most suitable AWS services and architecture to solve this problem efficiently. Are there any services or cloud architectures that can solve my problem?

Answer

We do not have a single service that will do it all, however, we do have EventBridge Scheduler, which allows you to create millions of schedules, either a cron, a rate based or individual schedules. You can create an application that checks the list every so often and creates schedules in EB Scheduler as needed. The application itself can be a Lambda function that is triggered by an EV Scheduler rate based schedule (e.g., every 5 minutes). Alternatively, if the list is stored in an S3 object, you can invoke the function every time the file is updates.

Answer

We have a similar system in place at [Rulation](https://rulation.com/) using a periodic EventBridge schedule invoking a lambda which dispatches jobs as needed. I'd recommend having a look at the following which details a possible implementation: [Serverless Scheduling with Amazon EventBridge, AWS Lambda, and Amazon DynamoDB](https://aws.amazon.com/blogs/architecture/serverless-scheduling-with-amazon-eventbridge-aws-lambda-and-amazon-dynamodb/).

How to use AWS services to run a function at specific scheduled times based on a provided list of date/time for an automation system?

Relevant content