Glue Interactive Python Jupyter notebook sessions

1

Hello.

I'm following the AWS docs here [https://docs.aws.amazon.com/glue/latest/dg/interactive-sessions-chapter.html] for developing Glue ETL pyspark jobs on my laptop using remote Glue Spark services in AWS. I get the errors below as soon as I follow the directions to start a session and attempt to actually use the session to do something. Is Glue still a supported service? I see that the documentation is quite old. Error message below.

Traceback (most recent call last):
  File "/Users/xxx/.asdf/installs/python/3.9.7/lib/python3.9/site-packages/aws_glue_interactive_sessions_kernel/glue_python_kernel/GlueKernel.py", line 95, in do_execute
    self.create_session()
  File "/Users/xxx/.asdf/installs/python/3.9.7/lib/python3.9/site-packages/aws_glue_interactive_sessions_kernel/glue_python_kernel/GlueKernel.py", line 518, in create_session
    self.set_session_id(self.glue_client.create_session(
  File "/Users/xxx/.asdf/installs/python/3.9.7/lib/python3.9/site-packages/botocore/client.py", line 657, in __getattr__
    raise AttributeError(
AttributeError: 'Glue' object has no attribute 'create_session'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/xxx/.asdf/installs/python/3.9.7/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 357, in dispatch_shell
    await result
  File "/Users/xxx/.asdf/installs/python/3.9.7/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 652, in execute_request
    reply_content = await reply_content
  File "/Users/xxx/.asdf/installs/python/3.9.7/lib/python3.9/site-packages/aws_glue_interactive_sessions_kernel/glue_python_kernel/GlueKernel.py", line 97, in do_execute
    self.ipython_display.send_error(f'Exception encountered while creating session: {e} \n')
  File "/Users/xxx/.asdf/installs/python/3.9.7/lib/python3.9/site-packages/hdijupyterutils/ipythondisplay.py", line 30, in send_error
    self._ipython_shell.write_err(u"{}\n".format(error))
AttributeError: 'ZMQInteractiveShell' object has no attribute 'write_err'

Here is my test code....

%iam_role arn:aws:iam::xxxxxx:role/service-role/AWSGlueServiceRole-dev
%session_id test-glue-session1
!jupyter --version
%list_sessions
%connections "dev-poc-glue-aurora-db"
sc = SparkContext()

and the output....

Welcome to the Glue Interactive Sessions Kernel
For more information on available magic commands, please type %help in any new cell.

Current iam_role is None
iam_role has been set to arn:aws:iam::xxxxxxxx:role/service-role/AWSGlueServiceRole-dev.
There is no current session.
Selected Jupyter core packages...
IPython          : 8.0.1
ipykernel        : 6.7.0
ipywidgets       : 7.6.5
jupyter_client   : 7.1.2
jupyter_core     : 4.9.1
jupyter_server   : not installed
jupyterlab       : not installed
nbclient         : 0.5.10
nbconvert        : 6.4.1
nbformat         : 5.1.3
notebook         : 6.4.8
qtconsole        : 5.2.2
traitlets        : 5.1.1
There is no current session.
Connections to be included:
dev-poc-glue-aurora-db
Authenticating with profile=default
glue_role_arn defined by user: arn:aws:iam::xxxxxxxxxxx:role/service-role/AWSGlueServiceRole-dev
Attempting to use existing AssumeRole session credentials.
Trying to create a Glue session for the kernel.
Worker Type: G.1X
Number of Workers: 5
Session ID: AROAJXDQBQMRZML4ALH7A:xxxxxxxxxx-8cff17de-6d96-4021-9f9c-5e6c9f3296b9

thanks so much for your assistance.

asked 2 years ago6701 views
3 Answers
1

Hello,

AWS Glue interactive session is currently in preview, the documentation is reflecting the steps that will need to be run once it is general available.

At the moment, you need still to run a manual step before you start the notebook the first time:

  • add the service model included with the install to botocore.

With your venv active run:

SITE_PACKAGES=$(pip3 show aws-glue-sessions | grep Location | awk '{print $2}')
cp $SITE_PACKAGES/aws_glue_interactive_sessions_kernel/service-2.json $SITE_PACKAGES/botocore/data/glue/2017-03-31/service-2.json

hope this helps

AWS
EXPERT
answered 2 years ago
  • I tried these steps, but in aws_glue_interactive_sessions_kernel folder, I do not have the service-2.json file.

0

Interactive Glue Sessions are now generally available. I made a step-by-step video tutorial on how to configure it to develop locally in pycharm for anyone who wants to follow along. https://youtu.be/04LMQxDxjGM

answered 2 years ago
AWS
EXPERT
reviewed 2 years ago
0

Hello, I have a similar issue. I follow the guidelines to install jupyter notebook (https://docs.aws.amazon.com/glue/latest/dg/interactive-sessions.html) in a WSL Ubuntu 22.04 and when I try create a session I got the message: Session failed to reach READY instead reaching terminal state FAILED. Current session is in an unhealthy terminal state. If you wish to create a new session use %stop_session before executing any further statements.

pip -V
pip 23.3.2 from /home/user/.local/lib/python3.10/site-packages/pip (python 3.10)

pip list
aws-glue-sessions         1.0.4
aws-requests-auth         0.4.3
boto3                     1.34.21
botocore                  1.34.21
jupyter                   1.0.0
jupyter_client            8.6.0
jupyter-console           6.6.3
jupyter_core              5.7.1
jupyter-events            0.9.0
jupyter-lsp               2.2.2
jupyter_server            2.12.5
jupyter_server_terminals  0.5.1
jupyterlab                4.0.10
jupyterlab_pygments       0.3.0
jupyterlab_server         2.25.2
jupyterlab-widgets        3.0.9

I tried installing directly on Ubuntu and on a virtualenv but I always get the same error.

Enter image description here

Enter image description here

Enter image description here

answered 2 months ago
  • Hello, after some investigation, the issue happens only when I add the «%extra_jars» and/or «%additional_python_models» option(s).

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions