Trigger Glue Crawler with S3 bucket event

Question

Hi , 
I would like to trigger Glue Crawler , once the S3 bucket specific prefix files are being added.  Is there any design pattern I can follow, seeking guidance from the experts.

I have gone through this blogpost : 
https://aws.amazon.com/blogs/big-data/run-aws-glue-crawlers-using-amazon-s3-event-notifications/

But I am looking for more simpler solution than extra resource creation. 
Thanks,

Accepted Answer

Hello.

Since the S3 event trigger cannot directly trigger the Glue crawler, you need to create a configuration using SNS or SQS as per the documentation you checked.  
https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html

I think it is also possible to run the Glue crawler using Glue workflow and EventBridge event using methods other than S3 event triggers.  
When using EventBridge event, there is no need to create SNS or SQS.  
https://docs.aws.amazon.com/glue/latest/dg/workflows_overview.html

Trigger Glue Crawler with S3 bucket event

Contenuto pertinente