I am getting very low speed while uploading data to AWS S3 deep archive storage class

0

I am getting very low speed while uploading data to AWS S3 deep archive storage class, same issue while single thread / session data copy and while multi thread / session data copy. Does AWS restrict upload bandwidth for users / per session / per bucket etc. Tried several ways through copy, rclone but no luck, any suggestions to enhance speed while uploading data to AWS S3. There is no issue with ISP, since other internet operation upload / downloads working as per expectations.

2 Answers
0

The slow upload speeds you're experiencing with AWS S3 Deep Archive storage class are not due to any specific restrictions imposed by AWS on upload bandwidth for users, sessions, or buckets. However, there are several factors that could be affecting your upload performance, and there are ways to optimize it.

First, it's important to note that Amazon S3 is designed to handle high request rates, capable of achieving thousands of transactions per second. The service automatically scales to accommodate high request rates, with a baseline of at least 3,500 PUT/COPY/POST/DELETE requests per second per prefix in a bucket.

To improve your upload speeds, consider the following suggestions:

  1. Use Amazon S3 Transfer Acceleration: This feature can enable faster transfers of files over long distances between your client and your Amazon S3 bucket. It's especially useful if you're uploading from a location far from the S3 region you're using.

  2. Implement parallel uploads: You can increase your write performance by parallelizing uploads. For instance, if you create multiple prefixes in an Amazon S3 bucket to parallelize writes, you could significantly scale your write performance.

  3. Utilize AWS DataSync: This service is designed for efficient transfer of large amounts of data into Amazon S3, potentially up to 10 times faster than open-source tools. It automatically handles many tasks, including optimizing network utilization.

  4. Consider using the AWS CLI with the Common Runtime (CRT) S3 client: The CRT implements automatic performance optimizations for applications using S3. It can provide significant speedups in many circumstances by automatically configuring based on the specifics of the instance type it's running on.

  5. Check your network conditions: While you mentioned there's no issue with your ISP, it might be worth running a network diagnostic to ensure there are no bottlenecks between your location and the AWS data center.

  6. Verify the file sizes: If you're uploading many small files, this can be slower than uploading fewer large files due to the overhead of each request.

If these suggestions don't improve your upload speeds, it would be advisable to contact AWS Support for a more in-depth investigation of your specific setup and requirements. They can provide tailored advice and potentially identify any underlying issues affecting your upload performance.
Sources
Introduction - Best Practices Design Patterns: Optimizing Amazon S3 Performance
Amazon S3 Features - Amazon Web Services
Accelerate Amazon S3 throughput with the AWS Common Runtime | AWS Storage Blog

profile picture
answered 13 days ago
0

There are no practical bandwidth limits in S3. You can flood a network connection of practically any size with a sufficiently parallelised upload, as long as your sending machine has the I/O and CPU capacity to keep up and your upstream network connection doesn't become a bottleneck.

To establish a clean baseline, I suggest you first configure your AWS CLI with the settings below by running these commands in the CLI:

aws configure set default.s3.max_concurrent_requests 16
aws configure set default.s3.max_queue_size 10000
aws configure set default.s3.multipart_threshold 128MB
aws configure set default.s3.multipart_chunksize 64MB
aws configure set default.s3.payload_signing_enabled false

Make sure that in the file .aws/config under your profile folder, the setting max_bandwidth is not included. If it's there, remove it and save the file before proceeding.

Test the upload performance first with those settings. If the throughput is low and your machine doesn't show excessive CPU utilisation or an I/O bottleneck, increase the max_concurrent_requests setting from 16 to 64 and try again. Does the throughput increase? If not, then it's almost certainly your internet connection or a local resource limit on your computer, such as your hard drive or network file share, that is slowing you down.

If throughput increased between 16 and 64 threads and your machine isn't showing a bottleneck, you can try increasing the settings further to 100 or 150 threads, for example. Depending on the size distribution of your files, you may also see some improvement by decreasing multipart_chunksize and multipart_threshold to 16 MB.

Scaling out transfers horizontally in this manner is how practically unlimited transfer rates can be achieved with S3, as long as the client platform and network connection can keep up. However, if your computer maxes out CPU or I/O, you'll typically get better performance by lowering the concurrency rate to what the machine can comfortably handle. With excessively high settings, it'll stall due to overloading the I/O system or CPU, decreasing throughput, even if it still stays relatively high.

In general, I'd also advise uploading data to the "Standard" storage class initially and then using an S3 lifecycle rule to transition them to the offline archive storage classes, such as Glacier Deep Archive. If you upload directly to the offline classes, it's easy to incur massive extra costs, for example by deleting files just uploaded, causing the remainder of the 180-day minimum charge for Deep Archive to be charged immediately (and not over 6 months) or by getting charged extra for storing objects so small that an S3 lifecycle rule wouldn't transition them to an offline Glacier class.

EXPERT
answered 12 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions