Does Data quality work with Apache Iceberg tables

0

I get an error when trying the recommended rules for an iceberg table: InvalidInputException (status: 400): Invalid table definition: Parameter "table_type" in table stg_l1_stf_lineitemmap is set to "iceberg" but storage location doesn't conform to the format "s3://<warehouse-dir>/<glue database>/<glue table id>". To specify your warehouse directory, pass it in the request through AdditionalOptions={WarehouseDirectory=<your-warehouse-dir>}

Also get this error when running DQ rule through a glue job python script that works for other tables: An error occurred while calling o81.getDynamicFrame. Reads and writes using Lake Formation permissions are not supported for iceberg tables.

neal_li
asked 5 months ago286 views
1 Answer
0

Please refer the below documentation, data quality is supported:
[1] AWS Glue Data Quality - General availability: new features - https://docs.aws.amazon.com/glue/latest/dg/glue-data-quality.html#data-quality-release-notes-ga

The following new features are available with the general availability of AWS Glue Data Quality:

  • Support for Apache Iceberg, Apache Hudi and Delta Lake
AWS
answered 5 months ago
  • Then why am i getting an error "An error occurred while calling o81.getDynamicFrame. Reads and writes using Lake Formation permissions are not supported for iceberg tables."

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions