Full Table Access (FTA) vs Fine-Grained Access Control (FGAC) for Apache Spark on AWS Lake Formation managed tables

Overview

Amazon EMR and AWS Glue Spark provides two modes for Apache Spark workloads that access tables in the AWS Glue Data Catalog that are governed by AWS Lake Formation:

• Full table access (FTA) is designed for jobs whose runtime role is authorized to access an entire table. Use FTA for trusted data-producer and table-management workloads that have/need unrestricted table-level access for operations such as CREATE, ALTER, INSERT, UPDATE, MERGE, DELETE, and DROP.

• Fine-grained access control (FGAC) evaluates Lake Formation permissions and data filters in an isolated system execution space. Use FGAC for data-consumer workloads when different users or roles require row-, column-, or cell-level restrictions.

Key Distinction

This distinction is important because a Spark cluster can be launched with only either of the configuration and not both. An FGAC job cannot run on the Spark cluster launched with FTA setting; while an FGAC cluster can still run a job that has access to the full table (reasonable).

Architecture Recommendation: Separate Producers from Consumers

There are few recommendations I would like to highlight for the Spark cluster configuration for your use-cases.

Try to separate your use-cases into those applications that need and utilize entire tables and those applications and use-cases that need row and column level access. Launch Spark clusters as two separate compute lanes – one with FTA configuration for producer/write workloads and one with FGAC clusters for read workloads. This separation is both a security boundary and a capacity-planning boundary. It prevents unrestricted producer roles from sharing an execution environment with restricted consumers, and it allows each lane to be sized for its own Spark driver and executor model.

1. Producer and Table-Management Lane (FTA)

• Dedicated EMR on EC2 clusters, EMR Serverless applications, or AWS Glue jobs
• Trusted runtime roles with full permissions on the tables they manage
• Batch ingestion, CDC merges, compaction orchestration, schema evolution, INSERT, UPDATE, MERGE, DELETE, and table lifecycle operations

2. Consumer Lane (FGAC)

• Separate EMR clusters or applications and AWS Glue jobs or sessions
• Runtime roles mapped to each consumer persona
• Interactive analysis, reporting, feature reads, and downstream transformations that require row-, column-, or cell-level restrictions

Why? The Modes Affect Spark Capacity and Cost

Refer to the diagram in this documentation.
FGAC cluster configuration isolates user code from Spark-system space, where the authorized data-access and filtering operations happen. This consumes additional driver and system-executor capacity and can hence add latency and cost of a larger cluster size compared with FTA cluster configuration.

FGAC Capacity Requirements

In FGAC Spark clusters, each job that touches Lake Formation managed tables, uses two Spark drivers—one user driver and one system driver. For pre-initialized capacity, you will need at least two drivers and, for best performance, twice as many drivers as a comparable job without FGAC. These driver workers and the user/system executor profiles consume cluster capacity for the entire duration of the job run.

AWS Glue 5.0+ FGAC documents at least four workers: one user driver, one system driver, one system executor, and one standby user executor. Four workers equal four DPUs (example numbers with G.1X workers).

Important Note

These FGAC and FTA distinctions matter only when your Data Catalog tables are governed by Lake Formation. If you are using AWS IAM based access control for Data Catalog, the IAM policies provide table level access control and not row/column level sharing options. You will use IAM policy permissions and S3 bucket policies to manage access for your EMR instance profile role/Glue job execution role. Spark configuration will not use any of the lakeformation properties for FGAC nor FTA.

Conclusion

Keep the governance boundary (FGAC) when it is required, but you do not need to pay the additional compute and transition cost for trusted producer jobs that already have unrestricted table permissions (FTA).

Resources

1. EMR on EC2 with FTA
2. EMR Serverless with FTA
3. AWS Glue with FTA
4. EMR on EC2 with FGAC
5. EMR Serverless with FGAC
6. AWS Glue with FGAC