How do I troubleshoot Spark applications that fail to run in my Amazon EMR notebook?

Short description

Spark applications that run in an Amazon EMR notebook might fail to start with the following error:

"The code failed because of a fatal error: Session 4 did not start up in 60 seconds."

Resolution

Check the resources on the cluster

Make sure that Spark has enough available resources in the cluster for Jupyter to create a Spark context. To check available resources use Amazon CloudWatch metrics or the Resource Manager.

Make sure that the Sparkmagic libraries are configured correctly

Contact your Jupyter administrator to make sure that the Sparkmagic libraries are configured correctly.

Restart the notebook kernel

Complete the following steps:

1. Open the Amazon EMR console.
2. In the navigation bar, choose Workspaces (Notebooks).
3. Select the workspace from the Workspaces (Notebooks) list, and then choose Quick Launch. Or choose the workspace from the list and choose Attach and then choose Launch in JupyterLab or choose Launch in Jupyter.
4. Choose Kernel, and then choose Restart Kernel.

Increase the Spark session timeout period for JupyterLab

Complete the following steps:

1. Open the Amazon EMR console.

2. In the navigation bar, choose Workspaces (Notebooks).

3. Select the workspace from the Workspaces (Notebooks) list, and then choose Quick Launch. A new browser tab opens to the JupyterLab editor.

4. Open the notebook terminal.

5. Run the following command to open the config.json file:

   vi /home/notebook/.sparkmagic/config.json

   Note: If you installed the JupyterHub application in the Amazon EMR primary instance, then run the following command:

   vi /etc/jupyter/conf/config.json

6. Add or update the livy_session_startup_timeout_seconds: ### option in the config.json file.

7. Restart all kernels.
   Note: If you installed the JupyterHub application in the Amazon EMR primary instance, then restart the JupyterHub container.

Tune Spark driver memory

Modify the Spark driver memory that the Jupyter notebook application uses to control the resource allocation. For more information, see How do I modify the Spark configuration in my Amazon EMR Studio Workspace?

Make sure that the Livy service is healthy

Check the status of the Livy server that runs on the primary node instance

Complete the following steps:

1. To check the status of the livy-server, run the following command:

   sudo systemctl status livy-server

2. If the server is status is down, then run the following command to restart the livy-server:

   sudo systemctl start livy-server

Increase the Livy Server memory

By default, the notebook client attempts to connect to Apache Livy server for 90 seconds. If the Livy server doesn't respond within 90 seconds, then the client generates a timeout. The most common reason why Livy server doesn't respond is lack of resources. To fix this, increase the memory for Livy server.

Complete the following steps:

1. Use SSH to connect to the primary node instance,.

2. Add the following property to the /etc/livy/conf/livy-env.sh file:

   export LIVY_SERVER_JAVA_OPTS="-Xmx8g" (option to your value)

3. Restart the Livy server.

   sudo systemctl stop livy-server; sudo systemctl start livy-server

Use cluster mode instead of client mode in Livy

When you submit Spark applications on the notebook in client mode, the Spark driver runs as the sub process of the Livy server. If you run a sub process, then you might experience a lack of resources on the primary node. To prevent Livy failures caused by insufficient resources, change the deployment mode to cluster mode. When you run Spark applications in cluster mode, the driver runs on the the core and task nodes, instead of the primary node.

Complete the following steps:

1. Use SSH to connect to the primary node..

2. Add the following parameter to the file /etc/livy/conf/livy.conf:

   livy.spark.deploy-mode  cluster

3. Restart the Livy server:

   sudo systemctl stop livy-server; sudo systemctl start livy-server