How do I troubleshoot the "java.lang.ClassNotFoundException" and related class loading errors in Spark on Amazon EMR?

8 minute read

My Apache Spark job on Amazon EMR fails with a "java.lang.ClassNotFoundException," "NoClassDefFoundError," "NoSuchMethodError," or "InvalidClassException" error."

Short description

The java.lang.ClassNotFoundException error, java.lang.NoClassDefFoundError, and java.lang.NoSuchMethodError are all class-loading failures. java.io.InvalidClassException is a class-version incompatibility error. These errors occur for one of the following reasons:

A required JAR is missing from the class path when the job runs.
There's a dependency version conflict. Your application bundles a different version of a library than the one that the Amazon EMR release provides, and the class loader resolves the wrong one.
A bootstrap action or custom configuration overrides the class paths, so the class loader picks up only the JAR files that exist in the location that you specified in your configuration.

To resolve the error, first check the stack trace for the name of the missing or conflicting class, and then determine which cause applies.

Resolution

Identify the missing or conflicting class

Find the first Caused by: entry in the stack trace and note the class name. Then determine which library provides that class and whether your application bundles its own copy.

If the class belongs to a library that your application uses but doesn't bundle, then the JAR is missing. To resolve this issue, see the Add a missing JAR section.

If the class belongs to a library that both your application and the Amazon EMR runtime provide, then the cause is a version conflict. To see which library and application versions your Amazon EMR release provides, see Application versions in Amazon EMR 7.x releases. To resolve this type of issue, see the Resolve dependency version conflicts section.

If the error appears after you add a bootstrap action or custom classification, then check whether it replaced the default class path. For more information, see the Check for overridden class paths section.

Add a missing JAR

It's a best practice to add a custom JAR at job submission with the --jars option. Amazon EMR automatically distributes JARs that you pass with the --jars flag to the cluster and includes them in both the driver and executor class paths.

To submit a Spark job with a custom JAR, run the following command:

spark-submit --deploy-mode client --master yarn \ 
   --class org.apache.spark.examples.SparkPi \ 
   --jars / PATH/TO/CUSTOM_JAR_1.jar,/ PATH/TO/CUSTOM_JAR_2.jar \ 
   spark-examples.jar 100

Note: Replace PATH/TO/CUSTOM_JAR_1.jar and PATH/TO/CUSTOM_JAR_2.jar with the paths to your JARs, and APPLICATION.jar with your application JAR.

Important: All spark-submit options, including --jars, must come before the application JAR. Spark passes anything after the application JAR to your application as an argument. Use a comma-separated list for multiple JARs. To prevent class conflicts, don't include JARs that already exist on the cluster. For more information, see Launching applications with spark-submit on the Apache Spark website.

On a running EMR on Amazon Elastic Compute Cloud (Amazon EC2) cluster, append your JAR path to spark.driver.extraClassPath and spark.executor.extraClassPath in /etc/spark/conf/spark-defaults.conf. Unlike --jars, the extraClassPath properties don't distribute files. The JAR must already exist at that path on every node.

Important: On Amazon EMR 6.0.0 and later, don't use the spark-defaults configuration classification to modify the extraClassPath properties. The value you provide replaces the long list of default JAR paths that Amazon EMR sets, it doesn't append them. And the default list changes between releases. Instead, pass the JAR with --jars, or use a bootstrap action that appends your path after the file exists.

The following bootstrap action appends your JAR path to the extraClassPath properties:

#!/bin/bash 
 # Appends /home/hadoop/extrajars/* to spark.*.extraClassPath 
 while [ ! -f /etc/spark/conf/spark-defaults.conf ] 
 do 
 sleep 1 
 done 
 sudo sed -i '/spark.*.extraClassPath/s/$/:\/home\/hadoop\/extrajars\/\*/' /etc/spark/conf/spark-defaults.conf 
 exit 0

Copy your JARs to that path on every node and confirm that the cluster's EC2 instance profile has permission to read the Amazon Simple Storage Service (Amazon S3) bucket that hosts the script. For more information, see Service role for cluster EC2 instances (EC2 instance profile).

Resolve dependency version conflicts

If the class belongs to a library that the EMR runtime also provides, then adding another copy of the JAR usually creates a second version conflict rather than resolving the error.
To resolve the version conflict, complete the following steps in order:

Note: Stop as soon as your job runs successfully. You don't need to complete every step.

Check the version that your Amazon EMR release bundles and align your application to it. On the cluster, check the JARs in /usr/lib/spark/jars.
Mark the provided dependencies as provided. When you build an assembly JAR with Maven or sbt, list Spark and Hadoop as provided dependencies. For example, <scope>provided</scope> in a Maven pom.xml, or provided in a sbt libraryDependencies entry so that your JAR doesn't bundle its own copies. The cluster supplies them at runtime. For more information, see Bundling your application's dependencies on the Apache Spark website.
Shade libraries that you must bundle. If your application needs a different version of a common library, relocate its packages so that your copy and the Amazon EMR-provided copy can coexist: with Maven, use the Maven Shade Plugin's relocation configuration (see Relocating Classes); with sbt, use an sbt-assembly shading rule (see Shading).
As a last resort, load your classes first. Set spark.driver.userClassPathFirst=true and spark.executor.userClassPathFirst=true to give your JARs precedence over Spark's. Both properties are experimental and spark.driver.userClassPathFirst works in cluster deploy mode only. If you're in client mode, then use --driver-class-path instead. For more information, see Spark configuration on the Apache Spark website.

Table format runtimes

For table formats, Amazon EMR releases bundle Iceberg, Hudi, and Delta runtimes. Don't supply your own copies of these runtime JARs on top of the bundled ones. Set the table format's classification and use the bundled version, or align your JAR to the bundled version.

Check for overridden class paths

If the error appears after you add a bootstrap action or custom configuration, then compare the effective class path with a default cluster. A custom spark-defaults classification or a script that rewrites spark-defaults.conf replaces the Amazon EMR-provided values. In the Spark UI Environment tab check spark.driver.extraClassPath and spark.executor.extraClassPath against an unmodified cluster of the same release. Then restore any Amazon EMR default paths that your customization dropped.

EMR Serverless

Amazon EMR Serverless has no cluster nodes to copy JARs to, you need to supply dependencies with the job run.

Depending on what you need to add, take one of the following actions.

Add a JAR

Upload to Amazon S3 and pass --jars s3://amzn-s3-demo-bucket/prefix/custom.jar or --conf spark.jars=... in the sparkSubmitParameters of StartJobRun. For more information, see Using Spark configurations when you run EMR Serverless jobs.

Add Python dependencies

Use --conf spark.submit.pyFiles=s3://.../deps.zip for individual files, or package a virtual environment and use --conf spark.archives=s3://.../pyspark_venv.tar.gz#environment with the PYSPARK_PYTHON environment variables. For more information, see Using Python libraries with EMR Serverless.

Add OS-level or pre-installed libraries

Build a custom image from the EMR Serverless base images. Add JARs to /usr/lib/spark/jars/ in the Dockerfile. For more informatino, see Customizing an EMR Serverless image.

Don't override --driver-class-path or the extraClassPath properties on EMR Serverless. Your value replaces the EMR-provided class path and that can cause errors such as ClassNotFoundException for com.amazon.ws.emr.hadoop.fs.EmrFileSystem. Use --jars instead. For more information see Using Spark configurations when you run EMR Serverless jobs.

Java 17 notes (Amazon EMR version 7)

On Amazon EMR version 7, Spark runs on Java 17 by default. The “java.lang.IllegalAccessError” or "InaccessibleObjectException" for JDK-internal classes such as sun.nio.ch.DirectBuffer look like missing classes but are Java version issues.

Spark 3.3.0 and later injects the required --add-opens JVM options automatically for standard spark-submit jobs. You receive these errors only when a custom launcher or JVM flags bypass Spark's defaults.

Driver and executors on different Java versions

On mixed clusters or clusters that use custom AMIs, the driver, executors, and YARN ApplicationMaster can run on different Java versions. To make every component use the same version, complete the following steps:

Choose the Java version for all components. For Amazon EMR versions 7.0 and later, use the default, Java 17.
In the spark-env classification, set JAVA_HOME to that version's path, such as /usr/lib/jvm/java-17-amazon-corretto.
On Amazon EMR 7.0 and later, set spark.executorEnv.JAVAHOME and spark.yarn.appMasterEnv.JAVA_HOME to the same path in spark-defaults.

For more information, see Configure applications to use a specific Java Virtual Machine.