Lambda does not trigger on all S3 file uploads

0

Good day,

I have a lambda function which is configured to trigger for all items created in a specific S3 folder. The event trigger config:

Bucket arn: arn:aws:s3:::dev-signal-bucket
Event types: s3:ObjectCreated:*
isComplexStatement: No
Notification name: has private info
Prefix: my_subfolder/

The first few lines of my lambda function src file handler:

def handler(event, context):
    time.sleep(10)
    try:
        temp_dir = tempfile.gettempdir()
        DOWNLOAD_PATH = os.path.join(temp_dir, 'data.npy')
        bucket = event['Records'][0]['s3']['bucket']['name']
        encoded_key = event['Records'][0]['s3']['object']['key']
        key = urllib.parse.unquote(encoded_key)
        print(f"Handling file: {key}")
        s3.download_file(bucket, key, DOWNLOAD_PATH)
    except Exception as e:
        print(f"Error downloading magnitude spectrum from S3: {e}")

Every 5 minutes, multiple measurements are uploaded to the S3 subfolder my_subfolder from a raspberry pi. Whenever a measurement is uploaded successfully to the S3 subfolder, the device logs the record_timestamp and the device id in a DDB table my-ddb-table. Most of the times, the upload triggers the lambda, which then calculates some values and updates the corresponding item in the DDB table with the calculated values. However, in some cases, the upload is successful, but my lambda function is not triggered. I do not even see the "Hanlding file: ..." in my Cloudwatch logs. I know the upload is successful because the IoT device logs in the DDB table and I can see these files in my S3 subfolder in the AWS console. When I re-upload one of these files manually in the console, the lambda triggers and calculates and stores the values correctly, meaning the files are not corrupted. Also, no errors are logged into my Cloudwatch logs.

My lambda does not throttle.

My question is: why is my lambda not always triggered by a successful S3 item upload?

Help is very much appreciated. If anything is unclear, please let me know. Thanks in advance!

Kind regards, Miranda

asked 7 months ago861 views
2 Answers
1
Accepted Answer

For quick troubleshooting, you could check the CloudWatch Metrics for your Lambda function and setting the graph for 1-minute intervals. Look at "Invocations" specifically to see if an invocation appeared to happen at all when a file was uploaded and no sign of the Lambda having executed was observed. Then look at the "Errors" and "Throttles" metrics to see if something is obviously failing.

Aside from the quick troubleshooting, a more robust way to construct this setup would be to have EventBridge filter the events from S3 and send them to your own SQS queue (backed by a dead-letter queue, DLQ). You could then set that custom SQS queue to trigger your Lambda. The built-in integration between SQS and Lambda will naturally take care of governing concurrency for your Lambda function, perform retries in a way that you can configure, and give you visibility and metrics into events arriving in SQS and how long they wait there to be processed. The DLQ for the custom SQS queue would also give you a reliable means to catch any completely failing invocations and hold them for up to 14 days for troubleshooting and possibly attempting to process again later.

EXPERT
answered 7 months ago
  • Thank you for the tips on the quick troubleshooting! This troubleshooting has been helpful, since it showed that the lambda seems to be invoked for every file that has been uploaded successfully. Also, no Errors and Throttles are observed.

    This pointed me to the root cause; another process running on the raspberry pi that creates items in DDB as well. In some cases, the lambda finishes storing its data faster than that process. Then that process overwrote the already-present item in DDB, causing it to miss data.

    Thanks again for your reply!

1

Hello.

Does it happen often that Lambda is not triggered?

The document below stated that it could take more than a minute to trigger.
So, is it possible that it's just taking time?
https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html

Amazon S3 event notifications are designed to be delivered at least once. Typically, event notifications are delivered in seconds but can sometimes take a minute or longer.

profile picture
EXPERT
answered 7 months ago
profile picture
EXPERT
reviewed 7 months ago
profile picture
EXPERT
reviewed 7 months ago
  • Hello, thank you for your reply!

    It happens ~5% of the successful uploads. The lambda does not trigger after some time, unfortunately, so it is not due to that. One of the successful uploads that did not trigger the lambda is 40minutes ago for example.

    Do you have any other suggestions?

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions