- Newest
- Most votes
- Most comments
The org.apache.spark.fgac.serialization.ClassAclException is triggered by EMR 7.x Fine-Grained Access Control (FGAC). FGAC enforces a strict Java class allowlist for serialized task results passed between executors and the driver. The Delta Lake and Iceberg plugins internally utilize java.util.concurrent.CopyOnWriteArrayList, which is currently missing from the EMR 7.13 FGAC allowlist, resulting in a security violation during the job execution.
Workarounds
Decouple Execution Graphs (Immediate Fix):
Do not execute a .union() across S3-path Delta tables and Glue-catalog Iceberg tables within the same operation. Read, transform, and write each source independently to intermediate Parquet storage on S3. Perform the .union() on the homogeneous Parquet files in a completely separate Spark action to break the lineage.
Standardize Catalog Access: Register the raw Delta tables in the Glue Data Catalog and configure Lake Formation permissions for them. Unifying both formats under standard catalog-managed access mitigates the mixed-context serialization issues under FGAC.
Engage AWS Support:
Open a ticket referencing the omission of the standard JDK class java.util.concurrent.CopyOnWriteArrayList from the EMR 7.13 FGAC allowlist. EMR support must provide a cluster-level patch or include the class in a future minor update.
Regarding your last comment: The reason the intermediate Parquet step still fails is likely that the global Spark extensions (org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions,io.delta.sql.DeltaSparkSessionExtension) remain active in your current Spark session. Because they hook deeply into the Catalyst optimizer, their internal processes continue to invoke the blocked java.util.concurrent.CopyOnWriteArrayList during task serialization for the .union() operation—even if you are strictly querying Parquet files at that stage.
To bypass this FGAC limitation, you need a true Hard Break by splitting the workload into two entirely separate Spark applications (two distinct spark-submit executions):
- Job 1 (Ingest) Run with the Delta and Iceberg extensions. Read the raw Delta/Iceberg tables and write them out as plain Parquet to an intermediate S3 path.
- Job 2 (Transform): Run WITHOUT the Delta and Iceberg extensions in your spark-submit configuration. Read the intermediate Parquet files, perform the .union() and transformations, and write the final output.
By completely removing the plugins from the context of the second job, Spark relies entirely on its native Parquet reader, breaking the problematic lineage and ensuring the EMR FGAC serialization block is never triggered.
I'm gonna change my application to split the job into two, thanks for the advice!
Hey Gabriel, I dug into this after seeing your post. The ClassAclException you're hitting is a documented FGAC platform restriction in EMR 7.10+. AWS explicitly blocks "Supply of additional jars for Spark extension, connector, or metastore" under FGAC (it's in the EMR Lake Formation Considerations page). The Iceberg and Delta extensions you're loading at spark-submit trigger this at the driver/executor serialization boundary, which is why even the intermediate Parquet step failed, the extensions stay active at the session level.
Your two-job workaround is the right move. To make it production-ready, you'll want to wrap it in a Step Functions state machine so Job 1 (extensions on, ingest to Parquet) and Job 2 (extensions off, union and transform) run sequentially with proper error handling and retries.
A couple things worth confirming: Are you running EMR 7.13 on EC2 or Serverless? And is the Delta source account genuinely separate from the Iceberg target account? If cross-account, the Lake Formation resource links and RAM v3 permissions need careful design or you'll hit Glue:GetTable Access Denied errors downstream.
answered 3 months ago
I put together a short breakdown of exactly this problem at https://cloudnestle.com/deck/wHEnNOgPgF7LDFvI.html if it's useful. For $100 I can deliver the full orchestration design doc: Step Functions or MWAA workflow definition for both jobs, IAM/Lake Formation role design
Relevant content
- AWS OFFICIALUpdated 2 years ago

So, i've tried to read the sources, save them as parquet on an intermediate path and then reading those parquet and proceeding with the transformations (unions and so on), but it did not work, the error i got was exactly the same when i tried the transformation with the direct source tables.