S3 File Gateway multipart upload

0

In AWS doc https://docs.aws.amazon.com/filegateway/latest/files3/best-practices-managing-multi-part-uploads.html S3 File Gateway will use multipart upload when transfer large files to S3. Do you have information from how many GB or MB, it will use multipart upload? I want to estimate the number of request if I upload file to S3 through file gateway. Because when use multipart upload, it cause many request

asked a year ago763 views
2 Answers
8

Greetings, S3 itself recommends using multipart uploads for files larger than 100 MB, and requires it for files larger than 5 GB. This information is from the general S3 documentation and not specifically about the S3 File Gateway, but it gives you an idea of when multipart uploads start to become useful or necessary.

It's important to remember that using multipart uploads does increase the number of requests, as each part is a separate request. The size of the parts can be anywhere from 5 MB to 5 GB, with the restriction that the last part can be smaller. So, for example, a 1 GB file could be uploaded as 200 parts of 5 MB each, which would result in 200 upload requests, plus one to initiate the multipart upload and one to complete it, for a total of 202 requests. But the same file could also be uploaded as 20 parts of 50 MB each, which would only require 22 requests.

If the primary concern is minimizing the number of requests, it would be better to use larger part sizes. However, using larger parts could potentially reduce upload performance if the network connection is not stable, as any issues would require reuploading a larger amount of data. It also reduces the ability to upload parts in parallel.

Please let me know if I answered your question.

Reference: Amazon S3 Multipart Upload Overview: https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html

profile pictureAWS
EXPERT
answered a year ago
profile picture
EXPERT
reviewed a year ago
  • Hi, Thank you for your answer! I have a concern abount the number of request when S3 File Gateway using multipart upload. I try to determine exactly when S3 File Gateway using multipart upload (from 100MB file or 5GB file) to estimate pricing, but as you said, it's not clear in the S3 File Gateway documentation. If you have information, please let me know Thank you so much

  • Thank you for your reply. I believe the switch over to MPU is at around 5GB. So if your file is closer to that Size, thats where the switch to MultiPart Upload will begin.

  • Greetings again, essentially the gateway might start using MPU earlier than 5GB if it would be a more efficient use of resources. However, any upload past 5GB will certainly use MPU. I hope that helps your question. let me know if it did.

  • Thank you for your information. I think I will estimate based on infor: S3 Gateway use MPU from 5GB. If you have more detail documentation, pleas share with me

  • Hi! Please let me know If I answered your question .

0

For more detailed information about multipart upload in Amazon S3, you can refer to the official AWS documentation. The following link provides an overview of multipart upload, including its benefits, considerations, and best practices:

Amazon S3 Multipart Upload Overview: https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html

This documentation page will provide you with comprehensive insights into how multipart upload works, its use cases, and guidance on effectively utilizing this feature in Amazon S3.

profile picture
answered a year ago
  • I understand how multipart uploads work when using CLI and we can choose use or not. But when use S3 File Gateway, it will automatically use multipart upload with large file. So I have a question: Which files will consider as large file in S3 File gateway

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions