How do I add Python packages with compiled binaries to my deployment package and make the package compatible with Lambda?

Short description

Python packages that contain compiled code, such as NumPy and pandas, aren't always compatible with Lambda runtimes by default. If you use pip to install these python packages, then the packages download and compile a module-name package for the architecture of the local machine. If you don't use a Linux operating system (OS), then this makes your deployment package incompatible with Lambda Python runtimes.

To make your deployment package or layer compatible with Lambda on a non-Linux OS, run the pip install command. When you run the command, use manylinux2014 as the value for the --platform parameter.

Note: macOS --platform tags don't work. For example, the win_amd64 and macosx_10_6_intel tags don't install a deployment package that's compatible with Lambda.

Resolution

The following steps are examples of how to install pandas for the Lambda Python 3.9 runtime that runs on x86_64 architecture.

1. Open a command prompt. Then, run the following pip command to confirm that your version of pip is version 19.3.0 or newer:

   pip --version

   If your version of pip is older than pip version 19.3.0, then run the following command to upgrade to the latest version of pip:

   python3.9 -m pip install --upgrade pip

2. Install the precompiled Python package's .whl file as a dependency in your Lambda function's project directory:

   Important: Replace my-lambda-function with the name of your function's project directory.

   pip install \        
       --platform manylinux2014_x86_64 \
       --target=my-lambda-function \
       --implementation cp \
       --python-version 3.9 \
       --only-binary=:all: --upgrade \
       pandas

3. Open your Lambda function's project directory. If you use macOS, then run the following command to open the project directory:

   cd my-lambda-function

4. In a text editor, create a new file named lambda_function.py. Then, copy and paste the following example code into the file:

   import numpy as np
   import pandas as pd
   def lambda_handler(event, context):
       df2 = pd.DataFrame(np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]]),columns=["a", "b", "c"])
       number = np.pi
       print(df2)
       print(number)

5. Save this file in your Lambda function's project directory.

6. Create a Lambda deployment package .zip file archive that includes all of the installed libraries and source code:

   zip -r ../my-deployment-package.zip 

7. Use the my-deployment-package.zip file archive to either create a new Python 3.9 Lambda function or to update an existing one. For instructions, see Working with .zip file archives for Python Lambda functions.

   Note: You can use a similar procedure to create a Lambda layer that can be used across multiple functions. For example, the following command creates a new Lambda layer to install pandas for the Lambda Python 3.9 runtime, that runs on arm64 architecture:

   pip install \    
       --platform manylinux2014_aarch64 \
       --target=./python/lib/python3.9/site-packages \
       --implementation cp \
       --python-version 3.9 \
       --only-binary=:all: --upgrade \
       pandas

Related information

Managing Lambda dependencies with layers