S3 Granular access

If you're ok with using folders (prefixes) for each user instead of individual files - then this link (https://aws.amazon.com/premiumsupport/knowledge-center/iam-s3-user-specific-folder/) might help.

NirvaShare could be a solution. Check the marketplace app at https://aws.amazon.com/marketplace/pp/prodview-hcqszucfxsvye

Normally I'd suggest that you start by looking at Bucket Policies because you can control access by user. But the bucket policy has a size limit (20kB) so you'll quickly run into issues there once you hit a reasonable number of users.

You don't say what authentication method you're using. Even though bucket policies aren't going to work, they assume that you're using IAM to authenticate and that is not going to scale given the number of users you're talking about.

Personally, I think you're going to have to build something custom as an overlay to S3 which authenticates the users and controls access; and I also think that you're going to need to use more than one bucket (maybe).

However, this is a detailed, nuanced and large topic. I'd strongly encourage you to reach out to your local AWS Solutions Architect to discuss what your options are here.

Edit April 2024: Check out this re:Invent video which talks to solving large scale data access problems with S3.