- Newest
- Most votes
- Most comments
A lot depends on the max size of the files and where they would be stored in the interim. Are they already persisted somewhere and you want to queue just some metadata?
Hello. I would recommend implementing retry logic if you get intermittent 500 error responses from Amazon S3. Each AWS SDK uses automatic retry logic and an exponential backoff algorithm. A great source to refer is How do I troubleshoot a HTTP 500 or 503 error from Amazon S3?
Refactoring to use Amazon SQS in the architecture might not work for your use-case due to SQS messages allowing a max 256 KiB size. To send messages larger than 256 KB, you can look into the Amazon SQS Extended Client Library for Java, though this library also sends an Amazon SQS message that contains a reference to a payload in Amazon S3.
Hope this helps.
Our file size very small it will take only 1.5kb its just used to store the event.
Relevant content
- asked 6 months ago
AWS OFFICIALUpdated a year ago- AWS OFFICIALUpdated 3 years ago
- AWS OFFICIALUpdated 4 years ago
- AWS OFFICIALUpdated a year ago
No It's not persisted somewhere. From our service will generate file and continuously sending those file to store in S3. But some months before we got error 500 internal server error returned from AWS S3. We analysed the logs and found the error happens from S3 server itself. So we are planning to implement some DQL technique to prevent this failure.
I'm guessing your max file size will be too big to fit in SQS messages (max 256KB). The usual solution for that is to queue just metadata and put the actual data in S3 ... which doesn't really help you here! If at all possible the best option would be to enhance the generating service to cope with S3 errors, retrying and waiting as needed. Otherwise you're going to need somewhere else to temporarily persist the files. EFS, which can be accessed from Lambda, might be a good option.
So there is no any DLQ mechanism available to prevent failure of AWS S3?
No there's not.