Why can’t I view the Apache Spark history events or logs from the Spark web UI in Amazon EMR?

Short description

You can view the Spark History Server on Amazon EMR with either of the following:

• Off-cluster access to persistent application user interfaces (EMR 5.25.0 and higher).
• On-cluster application user interfaces with a web proxy through an SSH connection.

Resolution

Off-cluster persistent Spark History Server or UI

Persistent Spark History Server events aren't accessible on EMR clusters that have the following:

• Multiple master nodes.
• EMR clusters integrated with AWS Lake Formation.
• The default directory was changed from HDFS to a different file system, such as Amazon Simple Storage Service (Amazon S3).

For more information, see Considerations and limitations.

Note: The persistent Spark History Server isn't suitable for load testing or for viewing thousands of Spark applications running in parallel. To load test or to view many applications, use the on-cluster Spark History Server.

On-cluster Spark History Server or UI

On-cluster Spark History Server events aren't accessible if you save the Spark events in an S3 bucket on Amazon EMR releases before 6.3* and 5.30. The Spark History Server in these Amazon EMR versions doesn't have the emrfs-hadoop-assembly JAR file required to access S3 buckets. Without this JAR file, you receive the following error when you try to access Spark History Server events:

INFO FsHistoryProvider: History server ui acls disabled; users with admin permissions: ; groups with admin permissions
Exception in thread "main" java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.amazon.ws.emr.hadoop.fs.EmrFileSystem not found

To avoid this error, use the following cluster configuration. This configuration adds the required emrfs-hadoop-assembly JAR file to the Spark History Server's classpath.

[
  {
    "classification": "spark-defaults",
    "configurations": [],
    "properties": {
      "spark.eventLog.dir": "s3://<yourbucket>/",
      "spark.history.fs.logDirectory": "s3://<yourbucket>/"
    }
  },
  {
    "classification": "spark-env",
    "configurations": [
      {
        "classification": "export",
        "properties": {
          "SPARK_DAEMON_CLASSPATH": "$(ls /usr/share/aws/emr/emrfs/lib/emrfs-hadoop-assembly-*)"
        }
      }
    ],
    "properties": {}
  }
]

In EMR 6.x releases (until 6.3) setting the Spark event log directory to S3 during cluster launch without the emrfs-hadoop-assembly causes cluster termination. If you make changes after the cluster is running, this can cause the configuration changes to revert.

Note: This workaround isn't necessary for Amazon EMR releases after 6.3 and 5.30 because the required JAR file is added by default to the folder /usr/lib/spark/jars/.

Keep in mind that the Spark History Server doesn't show the applications in the incomplete application list when Spark events are written to S3. Also, If the Spark context is improperly closed, the event logs don't properly upload to S3.

Cluster that's in a private subnet

If you use a private subnet for your cluster, make sure that the Amazon Virtual Private Cloud (Amazon VPC) endpoint policy for the private subnet is correct. The VPC endpoint policy must include arn:aws:s3:::prod.MyRegion.appinfo.src/* in the resource list of the S3 . For more information, see Minimum Amazon S3 policy for private subnet.

"NET:ERR_CERT_COMMON_NAME_INVALID" error

If you get the error  "NET:ERR_CERT_COMMON_NAME_INVALID" after you turn on on encryption in transit on the Spark UI, there is a browser certificate validation issue.

If you use Mozilla Firefox, then you see an option to accept the risk and continue using the certificate. If you're using Google Chrome, skip certificate validation. To skip, enter thisisunsafe on the warning page.