Usando AWS re:Post, accetti AWS re:Post Termini di utilizzo
AWS re:Postre:Post

S3 Events for burst uploads

1

Hi,

We are planning to ingest approx 2 lakh small pdf files(approx 40 kb) in burst mode from on Premise to Cloud(S3) on daily basis through AWS Datasync. We want to keep track of the files ingested on daily basis and keep updating the processing status as it gets processed.

Can someone suggest the best way to achieve it ? We initially thought of using S3 events to call Lambda to insert record in Dynamo DB table but since it may be a burst of files coming into same S3 bucket it might have throttling issue on Dynamo DB or may hit Lambda limits for the Account.

Please let me know the best way we can implement this.

Thanks in Advance.

Regards, Dhaval Mehta

Argomenti
ArchiviazioneMigrazione e modernizzazioneServerlessComputazionaliDatabase
Tag
Servizio di archiviazione semplice di AmazonAWS DataSyncAWS LambdaAmazon DynamoDB
Lingua
English
Dhaval Mehta
posta 2 anni fa359 visualizzazioni
4 Risposte
2
Risposta accettata

There are multiple ways to achieve this. But it really depends what kind of status you want to track.

For real timenotifications within a short duration, you can take advantage of Event notification from object upload and pair it with SNS/SQS/Lambda and use some sort of Database or a text/csv file to keep track of objects uploaded. This will require code.

If the tracking can wait for 24+ hours, then S3 inventory is a better choice. It's very easy to configure without any code.

AWS
rePostfanclub
con risposta 2 anni fa
0

Decouple the process by sending the S3 notification to a SQS queue. https://docs.aws.amazon.com/AmazonS3/latest/userguide/how-to-enable-disable-notification-intro.html

Then have your Lambda function poll the queue. https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html

profile pictureAWS
ESPERTO
kentrad
con risposta 2 anni fa
0

You could also look at using the S3 inventory https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html

profile picture
Robin Ford
con risposta 2 anni fa
0

I would say that even going with Lambda notification should be ok for you if the lambda is only saving to DynamoDB because it should be super fast. It's not so easy to reach the account limit. But it is possible, so you can notify SQS instead of Lambda and trigger Lambda from SQS but using batch. For standard SQS it can be maximum 10000 items in one batch, so you will call Lambda much less often.

profile picture
MG
con risposta 2 anni fa

Accesso non effettuato. Accedi per postare una risposta.

Una buona risposta soddisfa chiaramente la domanda, fornisce un feedback costruttivo e incoraggia la crescita professionale del richiedente.

Linee guida per rispondere alle domande

Contenuto pertinente