A lambda failed to start with the error "cannot import name 'is_s3express_bucket' from 'botocore.utils'"

Question

I recently encountered an issue with a newly deployed Lambda function running on Python 3.8 (version 3.8.v33). When attempting to execute boto3.client("s3") or boto3.resource("s3"), it throws an error stating: "cannot import name 'is_s3express_bucket' from 'botocore.utils'."

Interestingly, when I set up a new Lambda function through the AWS Console, this error does not occur. This leads me to believe the issue isn't with the code itself, as it seems to be conditionally reproducible.

Could anyone suggest approaches to isolate and identify the underlying cause of this issue?

**Minimum reproducing code I think**
```python
import boto3
s3 = boto3.resource("s3")
# s3 = boto3.client("s3")  # Same as above
```

**Reproduced environment**
> INIT_START Runtime Version: python:3.8.v33	Runtime Version ARN: arn:aws:lambda:ap-northeast-1::runtime:353a31d9fb2c7cac8474d278a6cf08824c7f87f698d61d1df2c128fc25a48d43

> boto3.__version__='1.26.90'
> botocore.__version__='1.29.90'

(added) region=ap-northeast-1

But I can't find any differences between the succeeded lambda and the failed lambda.

**What I have done**
* I saw the lambda run successfully at about 1:00 AM JST(+9:00)
* I deployed the stack including the lambda at 8:00 AM JST, and then I found the lambda failed to start.
  * I modified the lambda from the console and tested it, and then the error was reproduced.
* I deleted the stack and deployed again at 10:40 AM JST, and then the lambda still failed.

captain · Accepted Answer

We faced a similar issue, resolved by setting the version of both boto3 and botocore in requirements.txt and shipping as part of the vendor dependencies.

What I think is going on, the Lambda function's default/built-in/baked-in dependencies of boto3 and botocore are mismatched, such that boto3 is using latest version (1.33.2) but botocore is using a previous version (1.33.1).

I tested this hypothesis and was able to reproduce the issue:

```
(venv5) Braverman@Local ~/Repos/apply-api (set-boto-version-1.33.1) $ pip list | grep boto
boto3           1.33.2
botocore        1.33.1
```

```
(venv5) Braverman@Local ~/Repos/apply-api (set-boto-version-1.33.1) $ python
Python 3.8.3 (default, Jul  8 2020, 14:27:55) 
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
s3 = boto3.resource("s3")
>>> s3 = boto3.resource("s3")
Traceback (most recent call last):
  File "", line 1, in 
...
ImportError: cannot import name 'is_s3express_bucket' from 'botocore.utils' (/Users/steven.braverman/Repos/apply-api/venv5/lib/python3.8/site-packages/botocore/utils.py)
```

However, if I match up the environments, things run smoothly:

```
(venv5) Braverman@Local ~/Repos/apply-api (master) $ pip list | grep boto
boto3           1.33.2
botocore        1.33.2
```

```
>>> import boto3
s3 = boto3.resource("s3")
>>> s3 = boto3.resource("s3")
>>> 
```

It is interesting that the new version of Lambda you deployed is not facing this issue. Deploying the new lambda function must have triggered a pip install -U on the lambda function's dependencies, which seems to not occur for already existing lambda functions.

daniel_fondeadora · Answer

The answer above is correct. We also faced this issue because we were packing `s3transfer` from our local development environment with the Lambda function, while `boto` and `botocore` were coming from the Lambda environment.

The solution for us was to either use all of the libraries from the runtime (built-in) or package all of them with the function code. Went for the first one and it worked.

captain · Answer

There seems to be something off with your versions.
Where are you printing those versions from?
What is odd is that the "s3express" method that is causing the exception is something that was added with the latest version of botocore/boto3 (on 2023.11.28) in version 1.33.2.

It would be strange that older versions of botocore/boto3 are complaining about "s3express" when "s3express" does not exist until 1.33.2
[https://github.com/boto/botocore/commit/af4fdde1a2e053f3d9d9d91e7ad13167388528ce
]()

A lambda failed to start with the error "cannot import name 'is_s3express_bucket' from 'botocore.utils'"

Relevant content