S3 Events for burst uploads

1

Hi,

We are planning to ingest approx 2 lakh small pdf files(approx 40 kb) in burst mode from on Premise to Cloud(S3) on daily basis through AWS Datasync. We want to keep track of the files ingested on daily basis and keep updating the processing status as it gets processed.

Can someone suggest the best way to achieve it ? We initially thought of using S3 events to call Lambda to insert record in Dynamo DB table but since it may be a burst of files coming into same S3 bucket it might have throttling issue on Dynamo DB or may hit Lambda limits for the Account.

Please let me know the best way we can implement this.

Thanks in Advance.

Regards, Dhaval Mehta

4 個答案
2
已接受的答案

There are multiple ways to achieve this. But it really depends what kind of status you want to track.

For real timenotifications within a short duration, you can take advantage of Event notification from object upload and pair it with SNS/SQS/Lambda and use some sort of Database or a text/csv file to keep track of objects uploaded. This will require code.

If the tracking can wait for 24+ hours, then S3 inventory is a better choice. It's very easy to configure without any code.

AWS
已回答 2 年前
0

Decouple the process by sending the S3 notification to a SQS queue. https://docs.aws.amazon.com/AmazonS3/latest/userguide/how-to-enable-disable-notification-intro.html

Then have your Lambda function poll the queue. https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html

profile pictureAWS
專家
kentrad
已回答 2 年前
0
profile picture
已回答 2 年前
0

I would say that even going with Lambda notification should be ok for you if the lambda is only saving to DynamoDB because it should be super fast. It's not so easy to reach the account limit. But it is possible, so you can notify SQS instead of Lambda and trigger Lambda from SQS but using batch. For standard SQS it can be maximum 10000 items in one batch, so you will call Lambda much less often.

profile picture
MG
已回答 2 年前

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南