AWS Lambda and Python for NLP/ML Tasks & CICD using CodeCommit, CodeBuild and Code deploy

Question

Our backend services heavily rely on Java and Kotlin, and we host them in AWS Lambda. However, when it comes to running machine learning projects, particularly those involving LLM/GenAI, we have observed limited support for Java/Kotlin libraries. Considering the extensive community support and comprehensive ML libraries available in Python, we are contemplating hosting some of our NLP/ML related tasks using Python in Lambda.

We lack experience in this area and would greatly appreciate guidance from the AWS community. Specifically, we would like to understand the development process, CI/CD using CodeCommit, CodeBuild, and CodeDeploy for Python in AWS Lambda. Furthermore, we are interested in incorporating containers into our development workflow and CI/CD pipeline, as certain Python libraries may exceed Lambda's library size limitations.

Any workshop links or demo videos or tutorials on this topic with CDK support will be super helpful. Thanks in advance.

Answer

Hi - Here are some resources for you to start

1. https://serverlessland.com/learn?services=lambda - Learn from a collection of resources around common topics, including blog posts, videos, workshops and other training. Choose a learning path from the list below to learn more.
2. [Building CI/CD pipelines for Lambda canary deployments using AWS CDK](https://catalog.us-east-1.prod.workshops.aws/workshops/5195ab7c-5ded-4ee2-a1c5-775300717f42/en-US)
3. [AWS Lambda workshop ](https://catalog.us-east-1.prod.workshops.aws/workshops/66943748-f648-4a43-8b44-da8cfcc53286/en-US)
4. [CDK example for Lambda ](https://cdkworkshop.com/30-python/30-hello-cdk/200-lambda.html)
5. [Building Lambda functions with Python](https://docs.aws.amazon.com/lambda/latest/dg/lambda-python.html)
6. [The Complete AWS SAM Workshop](https://catalog.workshops.aws/complete-aws-sam/en-US)

Lambda layers provide a convenient way to package libraries and other dependencies that you can use with your Lambda functions. Using layers reduces the size of uploaded deployment archives and makes it faster to deploy your code.

AWS Lambda and Python for NLP/ML Tasks & CICD using CodeCommit, CodeBuild and Code deploy

Relevant content