S3 query file name without database


hi, I'm have this question. I need to create a document portal to download product manuals. The number of files is very high, we're talking about approximately 200,000 files totaling around 100GB in size. The files are stored in an S3 bucket. My portal should display the download link for all files whose file name contains a serial number. i've multiple manual for every product that in the name contain serial number. I'd prefer not to use a supporting database. My application executes a LIST operation on the objects to search for it, which impacts the S3 LIST costs significantly.

What's the best approach in these cases? Please consider also that the files are updated six times a day.

1 Answer
  • Thanks Antonio, I know about presigned URLs... I'm sure I'll use this solution or CloudFront because it's more secure. My question is how to optimize the S3 LIST because on 200.000 of file every LIST is expensive. It scans through every list to filter by file name... The output isn't extensive, around 10 results... but it scans through everything to return them. The ideal scenario would be a supporting inventory file where the application can query without much economic effort. There's S3 Inventory, but it seems it can only update once a day. I need it to update six times

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions