calculating time for migrating huge amount of data using S3 batch replication

Question

I need help to create an estimate of the time it takes to migrate 200 TB of data from account A s3 bucket to account B s3 bucket and some suggest some measures to reduce migration time.

Details:-
Total data size = 200 TB
Total no. of objects = 50 M
Average object size = 60 MB
bucket region = both buckets are in the same region (Tokyo)
service quota for Replication transfer rate(s3) = 1 GBPS
No. of replication rules in the source bucket = 2

I got this estimate from another repost:-
200 TB -> 200'000 GB. At 10 bits per byte content (TCP headers, other overhead), you will have transfer 200'000 GB x 10 bit/byte = 2'000'000 Gb. So, at 1 Gbps, 2'000'000 seconds will be needed. So, around 23 days and 6 hours.

**Does no. of objects and object size not matter in calculation?**
since there is a limit for put and get request in s3.
Amazon S3 supports a request rate of 3,500 PUT, COPY, POST, DELETE or 5,500 GET, HEAD requests per second per prefix in a bucket.

**Can you suggest me some measures to reduce the migration time.**
like if I increase the service quota to 10 GBPS will the  migration time reduce

Answer

Hi, 
2'000'000s at 1 GBps to migrate your volume give you the opportunity to execute 2M x 3'500 req/s = 7B requests per type of requests (PUT, COPY, etc.) over the total duration. Given the fact that you have 50M objects, it gives you the opportunity to execute 100+ requests of each type during the transfer. It should not be the bottleneck.

The  "at 1 Gbps" means that you can configure you replication setup to sustain this throughput during the whole transfer.

A key aspect will be to optimize properly the S3 cp or perf params: see https://docs.aws.amazon.com/cli/latest/topic/s3-config.html for all the options that you have in defining your optimal parallelism

Hope it helps!

Didier

calculating time for migrating huge amount of data using S3 batch replication

Relevant content