Does Data quality work with Apache Iceberg tables

0

I get an error when trying the recommended rules for an iceberg table: InvalidInputException (status: 400): Invalid table definition: Parameter "table_type" in table stg_l1_stf_lineitemmap is set to "iceberg" but storage location doesn't conform to the format "s3://<warehouse-dir>/<glue database>/<glue table id>". To specify your warehouse directory, pass it in the request through AdditionalOptions={WarehouseDirectory=<your-warehouse-dir>}

Also get this error when running DQ rule through a glue job python script that works for other tables: An error occurred while calling o81.getDynamicFrame. Reads and writes using Lake Formation permissions are not supported for iceberg tables.

neal_li
gefragt vor 6 Monaten316 Aufrufe
1 Antwort
0

Please refer the below documentation, data quality is supported:
[1] AWS Glue Data Quality - General availability: new features - https://docs.aws.amazon.com/glue/latest/dg/glue-data-quality.html#data-quality-release-notes-ga

The following new features are available with the general availability of AWS Glue Data Quality:

  • Support for Apache Iceberg, Apache Hudi and Delta Lake
AWS
beantwortet vor 6 Monaten
  • Then why am i getting an error "An error occurred while calling o81.getDynamicFrame. Reads and writes using Lake Formation permissions are not supported for iceberg tables."

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen