Single CRAWLER Table for S3 Path

Question

Hi Team,

I want to create only 2 crawler tables (Folder_x & Folder_y) for below path irrespective of the schemas

1. S3> Folder_x> Folder_x1>Folder_x2 > file1.txt 
2.  S3> Folder_x> Folder_x1>Folder_x2 >file2.txt
2. S3> Folder_y> Folder_y1>Folder_y2 > file3.txt 
3.  S3> Folder_y> Folder_y1>Folder_y2 >files4.txt

Could you please help me on this

Thank you in advance

Answer

You can follow the steps outlined at [Tutorial: Adding an AWS Glue crawler](https://docs.aws.amazon.com/glue/latest/ug/tutorial-add-crawler.html) and choose the S3 path as `S3> Folder_x` for the first one and for the second crawler choose path `S3> Folder_y`.

This will allow you to run or re-run each crawler independently as your data changes.

Single CRAWLER Table for S3 Path

Relevant content