How to use NLTK files and AWS Lambda

Question

Hi,

I am deploying a lambda function that utilizes the NLTK packages for preprocessing text. For the application to work I need to download the stop words, punkt and wordnet libraries. I have deployed using a docker image and SAM cli. When the function runs on AWS, I get a series of errors when trying to access the NLTK libraries.

The first error I got was that '/home/sbx_user1051/' cannot be edited. After reading solutions on stack over flow, I was pointed in the direction of needing to store the NLTK libraries in the /tmp/ directory because that is the only directory that can be modified.

Now, after redeploying the image with the changes to the code, I have the files stored in temp, but the lambda function does not search for that file when trying to access the stop words. It still tries to search for the file in these directories:

- '/home/sbx_user1051/nltk_data'
- '/var/lang/nltk_data'
- '/var/lang/share/nltk_data'
- '/var/lang/lib/nltk_data'
- '/usr/share/nltk_data'
- '/usr/local/share/nltk_data'
- '/usr/lib/nltk_data'
- '/usr/local/lib/nltk_data'

What should I do about importing the NLTK libraries needed when running this function on aws lambda?

Answer

Hi,

I encountered the same problem with the same error messages, and I was able to fix the issue by following these steps:

1) In the directory where your Python code application is located, create a new directory.

$ mkdir package

2) Run the following command to install the required libraries for your Python code:

$ pip install --target ./package

Example: $ pip install --target . /package   nltk

3) Invoke the Python interpreter with the following command:

$ python3

import nltk

4) Download the required nltk modules for your application:

Example:
 nltk.download("punkt")
 nltk.download("stopwords")

5) Quit the Python interpreter:

quit()

6) Create a directory named "nltk_data"  under   /package:

$ mkdir  nltk_data

7) Copy the nltk data to the directory /package/nltk_data:

$ cp -R /home/user/nltk_data/*    ./nltk_data

Note: Replace "/home/user" with your home directory path.

8) Create a .zip file with the installed libraries and nltk_data:

$ cd package

$ zip -r ../my_deployment_package.zip  .   (there is a dot at the end)

9) Add your lambda_function.py (your application) file to the .zip file:

$ cd ..

$ zip my_deployment_package.zip lambda_function.py

10) Create an AWS Lambda function on AWS and upload the file "my_deployment_package.zip." Pay attention to the following configuration options:

a) Configuration -> General configuration -> Timeout (choose the needed value to run the application).

b) Configuration -> General configuration -> Memory (choose the needed value to run the application).

c) Configuration -> General configuration -> Ephemeral storage (choose the needed value to run the application).

d) Configuration -> Environment variables -> Key = NLTK_DATA, Value = ./nltk_data

I hope it helps!

How to use NLTK files and AWS Lambda

Relevant content