Glue Crawlers and MongoDB Atlas

It is not possible to use a wildcard in the include path when crawling MongoDB data using AWS Glue crawlers. The include path for MongoDB must specify the exact database and collection names.

However, there are a few options to consider:

• Create multiple crawlers - one for each database/collection combination. This allows crawling specific subsets of data.
• Use an exclude pattern to skip certain collections or documents based on patterns. For example, to exclude any collection starting with "temp_", you can use the exclude pattern "temp_*".
• If the databases and collections have a consistent naming pattern, you can programmatically generate the crawler configuration including all the combinations. Then update this configuration periodically as needed.
• Another alternative is to copy the MongoDB data into Amazon S3 using tools like AWS Database Migration Service. You can then crawl the data in S3 using wildcards in the include path.