How to handle long running process (such as processing of 40GB video file) using lambda function?

Question

Hi There :-),

**Context:**
1. Two files will be uploaded to s3,
1.2. around 40GB avi format video file
1.2. around 100mB txt file
2. Whenever there's an upload happens, I want to pick those both files and apply some transformation logic (python code that takes these files as inputs - I already have these).
3. So, Created a Lambda with Python 3.10 runtime

**Explanation:**
Assume, The upload happens for 10KB avi file and 7MB txt file, it will trigger the lambda function, below are the steps am following,
1. download the avi into lambda environment from s3
2. apply transformation logic
3. store it in another path of lambda execution environment
4. video stored in #3 will be picked by another transformation logic and produce a CSV file
Now, When I upload 1GB avi video file, the first step of transformation logic takes more than 15minutes. So, lambda execution times out.

**FYI:**
for storing the large files (downloading from s3 as mentioned in explaination #1), I connected lambda function with EFS. And that EFS is mounted on EC2.

**Questions:**
1. So, How do I get the long running process running using lambda?
2. I would like to increase the timeout of lambda function to 5Hrs. Can I get help from AWS team and get the timeout increased to 5Hrs?
3. Or is there any other ways to achieve this?

Thanks.

Answer

> So, How do I get the long running process running using lambda?

Lambda cannot be used for processes that run for more than 15 minutes.  
So I think we need to consider other AWS services.  
Examples include AWS Batch and ECS.  
We could also consider AWS Glue, which is suited for ETL and other processes.  
https://docs.aws.amazon.com/batch/latest/userguide/what-is-batch.html  
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html  
https://docs.aws.amazon.com/glue/latest/dg/what-is-glue.html

> I would like to increase the timeout of lambda function to 5Hrs. Can I get help from AWS team and get the timeout increased to 5Hrs?

Unfortunately, contacting AWS support will not create a Lambda that will run for more than 15 minutes.

> Or is there any other ways to achieve this?

Please consider services other than Lambda here as explained above.  
In this case, you may want to use AWS Glue since you already have the Python code.  
https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-python.html

Answer

As stated, Lambda functions can only run for up to 15 minutes. There is no way to extend that.

Your options are:
1. Use a different service such as ECS Fargate.
2. Break the processing into multiple, smaller chunks (not always possible), e.g., take the large file, break it into a few smaller files. Then a Lambda function can processes each chunk. You will use Step Functions to orchestrate the entire process.

How to handle long running process (such as processing of 40GB video file) using lambda function?

Relevant content