AWS Glue Version 4 Jupyter Notebook- how to update to latest boto3?

Question

I am using Glue Version 4 notebook in Glue Studio. Also tried Script version in the console. 
All of them do not recognize Lake Formation hybrid opt-in APIs. Throws below error. 
"AttributeError: 'LakeFormation' object has no attribute 'list_lake_formation_opt_ins' "

Tried the below options:
1/ 
%additional_python_modules s3://aws-glue-assets-430431572129-us-east-1/scripts/boto3-1.34.144-py3-none-any.whl

Errors that 
"Failed to update job
[gluestudio-service.us-east-1.amazonaws.com] updateJob: InvalidInputException: Deprecated argument(s) used. Use "--pip-install" instead of "--additional-python-modules""

2/ 
When I force run the code, it still throws the error:
"AttributeError: 'LakeFormation' object has no attribute 'list_lake_formation_opt_ins'"
And boto3 versions seems to be the older 1.26.133

How do we upgrade the boto3 version  to the latest in the Glue Version 4 studio notebooks? Is this even possible?

aarts · Accepted Answer

This is the config that worked for the Glue Ray notebooks Glue Version 4:

```
%%configure
{
    '--additional-python-modules': 'boto3==1.34.144',
    '--python-modules-installer-option': '--upgrade'
}
```

Just a note, this has to be added in its own cell without any extra magics and has to be executed before you start your session:

```
import boto3
print(boto3.__version__)
```

Sachin_KV · Answer

You can use **%%configure** or **%additional_python_modules** [1] magic to upgrade boto3 version. This needs to be done at the very beginning i.e. before a session is created -

```
%%configure
{
"--additional-python-modules":"boto3==1.34.144"
}
or
%additional_python_modules boto3==1.34.144
```

This version of boto3 has lakeformation list_lake_formation_opt_ins API.

[1] https://docs.aws.amazon.com/glue/latest/dg/interactive-sessions-magics.html#interactive-sessions-supported-magics

AWS Glue Version 4 Jupyter Notebook- how to update to latest boto3?

Relevant content