How to Read Cross-Account Shared Hudi Tables via Lake Formation in AWS Glue 5.0

Overview

Reading Apache Hudi tables across AWS accounts using Lake Formation and AWS Glue 5.0 is a powerful pattern for data mesh and multi-account architectures. However, it requires precise configuration, especially when fine-grained access control (FGAC) is enabled. This guide walks through the common pitfalls, root cause, and the complete setup needed for seamless cross-account Hudi access.

Common Error When Accessing Shared Hudi Tables

When running an AWS Glue 5.0 job to read a cross-account shared Hudi table with FGAC enabled, you may encounter the following error:

Error Category: UNCLASSIFIED_ERROR; An error occurred while calling o168.count.
You don't have permission to access S3 data files. If you registered the table
with AWS Lake Formation, update the S3 permissions on the IAM role that you used
when you registered the table. Otherwise, update the S3 permissions for the
execution role that you used for the Spark job.

Root Cause

When FGAC is enabled, Lake Formation uses a credential-vending mechanism to provide temporary, scoped credentials for accessing S3 data files. Hudi must use the credential-aware storage class (HoodieCredentialedHadoopStorage) to work with these vended credentials.

However, if the Hudi catalog is not properly configured, Hudi falls back to the default HoodieHadoopStorage class, which attempts to access S3 directly using the job's IAM role bypassing Lake Formation's credential-vending entirely. This results in the permission error above.

The stack trace typically shows:

org.apache.hudi.storage.hadoop.HoodieHadoopStorage.listDirectEntries(HoodieHadoopStorage.java:196)
org.apache.hudi.metadata.FileSystemBackedTableMetadata.lambda$getPartitionPathWithPathPrefixUsingFilterExpression$8d4dc07c$1(FileSystemBackedTableMetadata.java:180)

This confirms that Hudi is using HoodieHadoopStorage instead of the expected HoodieCredentialedHadoopStorage.

Solution

Step 1: Set Glue Job Parameters

Add the following job parameters to enable Lake Formation FGAC and Hudi support:

--enable-lakeformation-fine-grained-access: true
--datalake-formats: hudi

These parameters instruct Glue to inject the credential-vending logic required for FGAC.

Step 2: Configure Spark Settings

Add the following Spark configuration to your Glue job:

--conf spark.serializer=org.apache.spark.serializer.KryoSerializer
--conf spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension
--conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog

Why this works: When the HoodieCatalog is properly registered as the Spark catalog, it routes all file access through Lake Formation's credential-vending mechanism. This ensures Hudi uses HoodieCredentialedHadoopStorage instead of the default HoodieHadoopStorage, allowing it to work with the temporary scoped credentials that Lake Formation provides.

Step 3: Read the Table

With the configuration in place, you can read the shared Hudi table:

df = spark.sql("SELECT * FROM my_database_rl.my_shared_hudi_table")
df.show()

Prerequisites for Cross-Account Access

Before the Glue job can read the shared table, ensure the following steps are completed:

1. Accept the resource share : In the consumer account, accept the shared resource via AWS Resource Access Manager (RAM).

2. Create a resource link : In the consumer account's Glue Data Catalog, create a resource link pointing to the shared database/table.

3. Grant Lake Formation permissions : Grant the Glue execution role in the consumer account the necessary Lake Formation permissions (SELECT, DESCRIBE) on the resource link and the target table.

4. IAM permissions on the Glue execution role : Ensure the execution role has:

   • lakeformation:GetDataAccess
   • glue:GetTable, glue:GetDatabase
   • Standard Glue job execution permissions

Summary

Configuration	Value
Glue Version	5.0
--enable-lakeformation-fine-grained-access	true
--datalake-formats	hudi
spark.serializer	org.apache.spark.serializer.KryoSerializer
spark.sql.extensions	org.apache.spark.sql.hudi.HoodieSparkSessionExtension
spark.sql.catalog.spark_catalog	org.apache.spark.sql.hudi.catalog.HoodieCatalog

References

• Using AWS Glue with AWS Lake Formation for fine-grained access control
• Apache Hudi with fine-grained access control
• AWS Lake Formation cross-account access