athena query issue with apache hudi

Question

recently i got this error in athena
Permission denied on S3 path: `s3://{S3_BUCKET}/{S3_PREFIX}/.hoodie`
where the location path that set in glue `s3://{S3_BUCKET}/{S3_PREFIX}/{DB_NAME}/{TABLE_NAME}/`

what i want to ask is why athena searching ".hoodie" directory under {S3_PREFIX}, where it should be inside the {TABLE_NAME} directory?
checked in the {S3_PREFIX} directory there no file/directory name ".hoodie" after the {S3_PREFIX}
and also check in emr-spark, with emr-spark i able to retrieve the data without getting permission denied issue.

Answer

Athena Uses Presto for DML queries and to query Hudi tables metadata for the hudi table is required which is store under ".hoodie/" folder inside the table location. So suppose if the hudi table is built upon S3 location "s3://mubucket/hudi/" then after creating a hudi table in this location a ".hoodie/" folder will be created.

Here in you case if the permission denied error is coming up for a different location other than table location if I understood it correct then, This can be due miss-configurations of the table. Hence can you try recreating another small sample hudi table and then run Athena queries over the same. Also check whether you have proper permissions over the bucket or not when running Athena queries.

athena query issue with apache hudi

関連するコンテンツ