How do I troubleshoot SageMaker AI notebook instance connectivity issues?

Short description

A SageMaker AI Jupyter notebook might be unresponsive or show errors for the following reasons:

• SageMaker AI can't establish a connection between Jupyter and the browser.
• The notebook kernel reached its defined timeout period.
• Your resource utilization load is high.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Troubleshoot connection issues between your Jupyter notebook and the browser

When you open a Jupyter notebook, you might receive the following error message:

"A connection to the notebook server could not be established. The notebook will continue trying to reconnect. Check your network connection or notebook server configuration."

To troubleshoot this issue, take the following actions:

• Restart your notebook instance. When you restart, the notebook instance moves to a new host. A restart can also resolve HTTP 503 and 504 errors in the browser.
  Note: It's a best practice to regularly restart notebook instances to keep notebook instance software updated.
• Restart your browser, clear your browser cache, or try a different browser.
• Use a different network connection.
• Check if the firewall, proxy, or antivirus software is blocking the connection.
• Check the log of all WebSockets in your browser. You can typically find this setting in the developer mode of the browser.
• Temporarily turn off all browser extensions, and then try again.

Generate a new notebook session token

The Jupyter notebook session token has a maximum validity of 12 hours. After the token expires, the session times out and you must refresh to reset the timeout token. However, the Jupyter kernel continues to run even if the browser disconnects.

To mitigate the effects of the 12-hour token, complete the following steps:

1. Write the results of your program to a file instead of to stdout.
2. Convert your program to a Python script, and then run it.
3. To generate a new URL with AuthToken, call the CreatePresignedNotebookInstanceURL API operation to generate a new URL with AuthToken. Then, enter the new URL in your browser before the session expires. This generates a new 12-hour session token.
4. Go directly to AuthorizedUrl.
5. (Optional) To open JuypterLab, modify the URL to add "view=Lab&" to the form: https://name>.notebook.sagemaker.aws?view=Lab&AuthToken=

Check your resource utilization

To check the resource utilization on your SageMaker AI notebook instance, run the following commands in the notebook terminal.

Memory utilization:

free -h

CPU utilization:

top

Disk utilization:

df -h

It's a best practice to use a lifecycle configuration script to publish the instance metrics to Amazon CloudWatch for visibility. For more information, see publish-instance-metrics on the GitHub website.

If there's high CPU, memory, or disk utilization, then restart the notebook instance and try again. Check that your SageMaker AI notebook instance type can support the configuration for your jobs. Change your instance type, if needed. To view a list of SageMaker AI instance types, see the Amazon SageMaker Notebooks section of the Amazon SageMaker pricing page.

Related information

How do I troubleshoot a SageMaker AI notebook instance that can't open Jupyter?

How do I resolve insufficient capacity errors when I launch my SageMaker AI resources?

Metrics collected by the CloudWatch agent on Linux and macOS instances