S3 Events for burst uploads

Question

Hi,

We are planning to ingest approx 2 lakh small pdf files(approx 40 kb) in burst mode from on Premise to Cloud(S3) on daily basis through AWS Datasync.  We want to keep track of the files ingested on daily basis and keep updating the processing status as it gets processed.

Can someone suggest the best way to achieve it ? We initially thought of using S3 events to call Lambda to insert record in Dynamo DB table but since it may be a burst of files coming into same S3 bucket it might have throttling issue on Dynamo DB  or may hit Lambda limits for the Account.

Please let me know the best way we can implement this.

Thanks in Advance.

Regards,
Dhaval Mehta

Accepted Answer

There are multiple ways to achieve this. But it really depends what kind of status you want to track.

For ~~real time~~notifications within a short duration, you can take advantage of Event notification from object upload and pair it with SNS/SQS/Lambda and use some sort of Database or a text/csv file to keep track of objects uploaded. This will require code.

If the tracking can wait for 24+ hours, then S3 inventory is a better choice. It's very easy to configure without any code.

Answer

Decouple the process by sending the S3 notification to a SQS queue. 
https://docs.aws.amazon.com/AmazonS3/latest/userguide/how-to-enable-disable-notification-intro.html

Then have your Lambda function poll the queue.
https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html

Answer

I would say that even going with Lambda notification should be ok for you if the lambda is only saving to DynamoDB because it should be super fast.
It's not so easy to reach the account limit. But it is possible, so you can notify SQS instead of Lambda and trigger Lambda from SQS but using batch. For standard SQS it can be maximum 10000 items in one batch, so you will call Lambda much less often.

Answer

You could also look at using the S3 inventory
https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html

S3 Events for burst uploads

Relevant content