AWS DataBrew - DeltaLake source

Question

Hello Team,

how can we read Delta Lake (delta parquet) source with DataBrew?
Currently,  I am getting an error when trying to run a job:

```
An error occurred while calling o104.getDynamicFrame. s3://[REDACTED]/manifest is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [117, 101, 116, 10]

```

The datatype of the dataset is Data Catalog table. With Athena working fine, can fetch the data.

Answer

Hello,

Glue Catalog table based on Delta Lake is not supported by Glue DataBrew at this moment. As a workaround, you can use Glue ETL jobs to read and write Delta Lake tables. For more details and examples, refer to: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-format-delta-lake.html

I hope this information helps.

AWS DataBrew - DeltaLake source

Contenuto pertinente