Trigger Glue Crawler with S3 bucket event

0

Hi , I would like to trigger Glue Crawler , once the S3 bucket specific prefix files are being added. Is there any design pattern I can follow, seeking guidance from the experts.

I have gone through this blogpost : https://aws.amazon.com/blogs/big-data/run-aws-glue-crawlers-using-amazon-s3-event-notifications/

But I am looking for more simpler solution than extra resource creation. Thanks,

1 réponse
0
Réponse acceptée

Hello.

Since the S3 event trigger cannot directly trigger the Glue crawler, you need to create a configuration using SNS or SQS as per the documentation you checked.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html

I think it is also possible to run the Glue crawler using Glue workflow and EventBridge event using methods other than S3 event triggers.
When using EventBridge event, there is no need to create SNS or SQS.
https://docs.aws.amazon.com/glue/latest/dg/workflows_overview.html

profile picture
EXPERT
répondu il y a 7 mois

Vous n'êtes pas connecté. Se connecter pour publier une réponse.

Une bonne réponse répond clairement à la question, contient des commentaires constructifs et encourage le développement professionnel de la personne qui pose la question.

Instructions pour répondre aux questions

Contenus pertinents