Lambda invoke with boto3

Question

I have a lambda function developed in python and using paramiko library to connect to external sftp server and download files.

When I am testing lambda inside AWS Lambda console, everything is working perfectly. However, if I invoke lambda from external system using boto3 lambda invoke functionality, I see in Lambda monitor and Cloudwatch logs, that after 1 minute of execution, there is second lambda instance started, and after 1 minute of second instance execution - third instance started. There is no failures in any of them, all 3 runs are successful. Boto3 returns me ID of the third execution.

After my investigation I figured out that there is one bigger file in sftp, which processing takes 70-80 seconds and every time when this file is being processed another lambda execution happens. I think maybe the problem is that boto3 lambda invoke is checking lambda status and during that period when this file is being processed, there is no heartbeat and invoke launches another instance?

I need to get lambda response to guarantee that all files were downloaded before launching further steps in my logic, but I don't want to run it 3 times every time.

Any ideas how can I overcome this issue?

Thank you.

====================================================================================================

UPDATE:

Seems that boto3 lambda invoke has default socket and connection timeout 60 seconds: https://repost.aws/knowledge-center/lambda-function-retry-timeout-sdk

I updated my code by changing default values to 900 seconds - max Lambda execution time: 
```    
client = boto3.client('lambda', region_name="us-east-1", config=Config(read_timeout=900, connect_timeout=900,  tcp_keepalive=True, retries={"max_attempts": 0}))

out = client.invoke(
        FunctionName=function_name,
        InvocationType='RequestResponse'
    )

payload = json.loads(out["Payload"].read())
logger.info(f"LAMBDA PAYLOAD: {payload}")

```

Now if I am running this peace of code from my computer it works, lambda gets completed ~7 minutes
After that I tried to run the same logic from MWAA, lambda gets triggered and from the logs I see it gets completed successfully, but task in airflow runs exactly for 900 seconds and throws error ``` TimeoutError: The read operation timed out```
Instead of downloading files I just executed sleep(300) with success. Then sleep(360) and it run for 900s until timeout.

I believe now I am facing this situation: https://aws.amazon.com/blogs/networking-and-content-delivery/implementing-long-running-tcp-connections-within-vpc-networking

I am stuck once again.

Accepted Answer

With Lambda Functions you have to consider:
1. Maximum amount of run duration time. (15 min). Make sure when creating the lambda function you set the timeout value. In the console look at the configuration and if you are creating the lambda function with code, "client.create_function", set the Timeout value to 15 min (900 sec)

2. How are you invoking the function? Make sure you are using the "RequestResponse" invocationType for Synchronous and Event for Async.
```
response = client.invoke(
    FunctionName='string',
    InvocationType='Event'|'RequestResponse'|'DryRun',
    LogType='None'|'Tail',
    ClientContext='string',
    Payload=b'bytes'|file,
    Qualifier='string'
)
```

3. Make sure you handle the Error in Try Except else. depending on error type, lambda executes up to two more times.

```
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/lambda/client/invoke.html
```

4. Your logic. is the function looping until all the files are downloaded? How about if there are 10, 100, or 1000 files to download?  The time required to download all the files may exceed the 15 min limit.

Answer

Not sure about your use-case. But, Keeping socket connections open for that long is not a good thing unless there is a good reason like real-time telemetry.

I would think you would want to send a message per file to a process SQS queue and have the lambda pickup file and process the file. Then when done processing, have the lambda send a message to a completed SQS queue.

In you calling app, just waits until there is something in the completed queue. Once all the files are done, you can continue.

Lambda invoke with boto3

Relevant content