- Newest
- Most votes
- Most comments
A simple approach could be to configure the S3 buckets to send event notifications to EventBridge, where a rule configured for the default bus for the region of the bucket would match PutObject events for the appropriate bucket or several, and send them to a lightweight Step Functions workflow or Lambda function that would simply update a per-bucket "last update" timestamp value in a DynamoDB table.
Secondly, configure a similar scheduled rule as you have currently but triggering another lightweight Step Functions workflow that would first check if the last update timestamp of the bucket is later than the last time the crawler job has been executed. If no changes have taken place, the Step Functions workflow would simply do nothing, without any significant costs, and otherwise it would update a "last executed" timestamp in the DynamoDB table and trigger the crawler job similarly as your current schedule is doing.
Relevant content
- asked 5 months ago
- asked 3 months ago
- AWS OFFICIALUpdated 3 years ago
- AWS OFFICIALUpdated 20 days ago
- How can I use a Lambda function to automatically start an AWS Glue job when a crawler run completes?AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 2 years ago