AWS DataBrew - DeltaLake source

Question

Hello Team,

how can we read Delta Lake (delta parquet) source with DataBrew?
Currently,  I am getting an error when trying to run a job:

```
An error occurred while calling o104.getDynamicFrame. s3://[REDACTED]/manifest is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [117, 101, 116, 10]

```

The datatype of the dataset is Data Catalog table. With Athena working fine, can fetch the data.

Ethan_H · Answer

Hello,

Glue Catalog table based on Delta Lake is not supported by Glue DataBrew at this moment. As a workaround, you can use Glue ETL jobs to read and write Delta Lake tables. For more details and examples, refer to: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-format-delta-lake.html

I hope this information helps.

AWS DataBrew - DeltaLake source

Relevanter Inhalt