Skip to content

EMR with FGAC, Delta Lake and Iceberg

0

Hi,

I'm working on a Spark Scala App that reads Delta Lake (with CDF) tables directly from S3 (not catalog registred) and writes a parquet table. The productive process is being executed on EMR 6.11 without Lake Formation. Those source Delta Lake tables are being migrated to Iceberg and being registred on a Glue Catalog (cross account), and now i have to refactor this app to read from these tables. Not all sources tables have been migrated, so i'm in a position where i need to read both Delta Lake via S3 path and Iceberg via catalog. I've been able to read all the source tables (Delta and Iceberg) and write them as Parquet on S3 for testing, but when i try to union and transform all tables at once, i'm getting a FGAC serialization error when getting executor result from the final write. I've tried to save all the sources as parquet and then reading them, because i though that it might been an "mixed format" error, but it did not solve the problem.

I'm using EMR 7.13 with Lake Formation and FGAC (it is needed, FTA does not solve my problem), and my cluster has the following configuration:

[
  {
    "Classification": "iceberg-defaults",
    "Properties": {
      "iceberg.enabled": "true"
    }
  },
  {
    "Classification": "delta-defaults",
    "Properties": {
      "delta.enabled": "true"
    }
  },
  {
    "Classification": "spark-hive-site",
    "Properties": {
      "hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory"
    }
  },
  {
    "Classification": "hive-site",
    "Properties": {
      "hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory"
    }
  },
  {
    "Classification": "spark-defaults",
    "Properties": {
      "spark.databricks.delta.changeDataFeed.timestampOutOfRange.enabled": "true"
    }
  }
]

My spark submit is something like this:

spark-submit
--deploy-mode cluster
--master yarn
--name spark_app_name
--conf spark.sql.session.timeZone=America/Sao_Paulo
--conf spark.scheduler.mode=FAIR
--conf spark.shuffle.service.enabled=true
--conf spark.dynamicAllocation.enabled=true
--conf spark.dynamicAllocation.minExecutors=1
--conf spark.dynamicAllocation.initialExecutors=10
--conf spark.dynamicAllocation.executorIdleTimeout=120s
--conf spark.dynamicAllocation.cachedExecutorIdleTimeout=300s
--conf spark.dynamicAllocation.schedulerBacklogTimeout=180s
--conf spark.default.parallelism=2400
--conf spark.sql.parquet.datetimeRebaseModeInWrite=CORRECTED
--conf spark.sql.parquet.int96RebaseModeInWrite=CORRECTED
--conf spark.sql.legacy.parquet.datetimeRebaseModeInRead=CORRECTED
--conf spark.sql.legacy.parquet.datetimeRebaseModeInWrite=CORRECTED
--conf spark.sql.legacy.timeParserPolicy=LEGACY
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions,io.delta.sql.DeltaSparkSessionExtension
--conf spark.sql.sources.partitionOverwriteMode=dynamic
--conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog
--conf spark.sql.catalog.spark_catalog.warehouse=s3://generic-path
--conf spark.sql.catalog.spark_catalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog
--conf spark.sql.catalog.spark_catalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO
--conf spark.sql.catalog.spark_catalog.glue.account-id={aws_account}
--conf spark.sql.catalog.spark_catalog.glue.id={aws_account}
--conf spark.sql.catalog.spark_catalog.client.assume-role.region={aws_region}
--conf spark.sql.catalog.spark_catalog.client.region={aws_region}
--conf spark.sql.secureCatalog=spark_catalog
--conf spark.sql.catalog.spark_catalog.lf.managed=true
--conf spark.driver.extraJavaOptions=\"-Daws.region={aws_region}\"
--conf spark.executor.extraJavaOptions=\"-Daws.region={aws_region}\"
--conf spark.sql.parquet.compression.codec=snappy
--conf spark.yarn.maxAppAttempts=1
--conf spark.sql.files.ignoreMissingFiles=true
--conf spark.yarn.heterogeneousExecutors.enabled=false
--conf spark.driver.cores=16
--conf spark.driver.memory=96G
--conf spark.executor.cores=32
--conf spark.executor.memory=128G
--conf spark.executor.instances=10
--conf spark.sql.shuffle.partitions=320
--files s3://generic-path/resources/task-conf.conf,iles s3://generic-path/resources/app-conf.conf,
--conf spark.executor.extraJavaOptions=-Dconfig.file=s3://generic-path/resources/task-conf.conf,s3://generic-path/resources/app-conf.conf,
--conf spark.driver.extraJavaOptions=-Dconfig.file=s3://generic-path/resources/task-conf.conf,s3://generic-path/resources/app-conf.conf,
--class com.company.package.Main https://nexus/repository/snapshots/com/company/package/app-jar.jar
--env prod
--task-name task_name_test
--app-param-1 s3://generic-path-raw
--app-param-2 s3://generic-path-stage
--app-param-3 s3://generic-path-analytics
--app-param-4 s3://generic-path-analytics/book/proccess/folder
--app-param-5 "2026-08-06 08:00:00"
--app-param-6 "2026-08-06 10:00:00"
--app-param-7 False

I know some confs on my spark-submit is conflicting, but i don't think it something related to the error i'm getting, which is:

ERROR SparkFGACExceptionSanitizer: Client received error with id = bca49ba6-5a1a-4d14-b95d-39c6c8ec0617, reason = SparkSecurityValidationException, message = Job aborted due to stage failure: Exception while getting task result: org.apache.spark.fgac.serialization.ClassAclException: Disallowed class cannot be resolved from the serialization. (java.util.concurrent.CopyOnWriteArrayList)
ERROR SparkFGACExceptionSanitizer: SECURITY ERROR. EXITING JVM.
org.apache.spark.fgac.error.SparkSecurityValidationException: Job aborted due to stage failure: Exception while getting task result: org.apache.spark.fgac.serialization.ClassAclException: Disallowed class cannot be resolved from the serialization. (java.util.concurrent.CopyOnWriteArrayList)

Can anyone help me understand why i'm getting this serialization error and how to solve? Thanks!

asked 3 months ago183 views

3 Answers
1

The org.apache.spark.fgac.serialization.ClassAclException is triggered by EMR 7.x Fine-Grained Access Control (FGAC). FGAC enforces a strict Java class allowlist for serialized task results passed between executors and the driver. The Delta Lake and Iceberg plugins internally utilize java.util.concurrent.CopyOnWriteArrayList, which is currently missing from the EMR 7.13 FGAC allowlist, resulting in a security violation during the job execution.

Workarounds

Decouple Execution Graphs (Immediate Fix): Do not execute a .union() across S3-path Delta tables and Glue-catalog Iceberg tables within the same operation. Read, transform, and write each source independently to intermediate Parquet storage on S3. Perform the .union() on the homogeneous Parquet files in a completely separate Spark action to break the lineage.

Standardize Catalog Access: Register the raw Delta tables in the Glue Data Catalog and configure Lake Formation permissions for them. Unifying both formats under standard catalog-managed access mitigates the mixed-context serialization issues under FGAC.

Engage AWS Support: Open a ticket referencing the omission of the standard JDK class java.util.concurrent.CopyOnWriteArrayList from the EMR 7.13 FGAC allowlist. EMR support must provide a cluster-level patch or include the class in a future minor update.

EXPERT

answered 3 months ago

AWS
SUPPORT ENGINEER

reviewed 3 months ago

  • So, i've tried to read the sources, save them as parquet on an intermediate path and then reading those parquet and proceeding with the transformations (unions and so on), but it did not work, the error i got was exactly the same when i tried the transformation with the direct source tables.

1

Regarding your last comment: The reason the intermediate Parquet step still fails is likely that the global Spark extensions (org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions,io.delta.sql.DeltaSparkSessionExtension) remain active in your current Spark session. Because they hook deeply into the Catalyst optimizer, their internal processes continue to invoke the blocked java.util.concurrent.CopyOnWriteArrayList during task serialization for the .union() operation—even if you are strictly querying Parquet files at that stage.

To bypass this FGAC limitation, you need a true Hard Break by splitting the workload into two entirely separate Spark applications (two distinct spark-submit executions):

  • Job 1 (Ingest) Run with the Delta and Iceberg extensions. Read the raw Delta/Iceberg tables and write them out as plain Parquet to an intermediate S3 path.
  • Job 2 (Transform): Run WITHOUT the Delta and Iceberg extensions in your spark-submit configuration. Read the intermediate Parquet files, perform the .union() and transformations, and write the final output.

By completely removing the plugins from the context of the second job, Spark relies entirely on its native Parquet reader, breaking the problematic lineage and ensuring the EMR FGAC serialization block is never triggered.

EXPERT

answered 3 months ago

AWS
SUPPORT ENGINEER

reviewed 3 months ago

  • I'm gonna change my application to split the job into two, thanks for the advice!

0

Hey Gabriel, I dug into this after seeing your post. The ClassAclException you're hitting is a documented FGAC platform restriction in EMR 7.10+. AWS explicitly blocks "Supply of additional jars for Spark extension, connector, or metastore" under FGAC (it's in the EMR Lake Formation Considerations page). The Iceberg and Delta extensions you're loading at spark-submit trigger this at the driver/executor serialization boundary, which is why even the intermediate Parquet step failed, the extensions stay active at the session level.

Your two-job workaround is the right move. To make it production-ready, you'll want to wrap it in a Step Functions state machine so Job 1 (extensions on, ingest to Parquet) and Job 2 (extensions off, union and transform) run sequentially with proper error handling and retries.

A couple things worth confirming: Are you running EMR 7.13 on EC2 or Serverless? And is the Delta source account genuinely separate from the Iceberg target account? If cross-account, the Lake Formation resource links and RAM v3 permissions need careful design or you'll hit Glue:GetTable Access Denied errors downstream.

answered 3 months ago

  • I put together a short breakdown of exactly this problem at https://cloudnestle.com/deck/wHEnNOgPgF7LDFvI.html if it's useful. For $100 I can deliver the full orchestration design doc: Step Functions or MWAA workflow definition for both jobs, IAM/Lake Formation role design

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.