By using AWS re:Post, you agree to the AWS re:Post Terms of Use

AWS Crawler to directly read Delta lake files from S3

0

Are there any ways to read delta lake files from s3 and create Data catalog on top of this to run Glue ETL job? When I crawl in delta folders it creates separate schema for log, manifest & parquets rather then each tables with all the delta log, manifest files and parquet files , https://docs.aws.amazon.com/glue/latest/dg/crawler-data-stores.html says Crawler now have native client then why it is not recognizing the path , Am I missing something.

asked 2 years ago620 views
1 Answer
0

Hi,

Please try installing the Delta Lake Connector for AWS Glue which can be found here.

https://aws.amazon.com/marketplace/pp/prodview-seypofzqhdueq

The Delta Lake Connector will allow you to connect to Delta Lake tables from your Glue jobs and it is offered at no additional cost.

Hope this helps.

profile pictureAWS
answered 2 years ago
profile picture
SUPPORT ENGINEER
reviewed 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions