- Newest
- Most votes
- Most comments
-
Is it possible to use s3 batch replication for the migration?
Yes, you can s3 batch replication for this much of data.
-
If i create 1 replication job for replicating the existing data, will all the replication configuration be used? if so how?
It's not clear, what do you mean by "all replication configuration"
-
How secure the replication of data will be?
Data in transit is encrypted, encryption at rest would fall into your responsibilities
-
What will be the bandwidth/throughput of data transfer?
Your network bandwidth won't come in to the picture, since it's All AWS. Most objects replicate within 15 minutes, but sometimes replication can take a couple hours or more. Refer this re:Post
-
How will the charges be generated?
Refer the S3 Replication Pricing
Additionally, I'd suggest you to go through this storage blog, which cover all the copy options and their pros/cons.
If you have not seen this before, I encourage you to look at the best practices on Batch Replication : https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-batch-replication-batch.html To answer your questions :
1- Yes, it is the method proposed by AWS so replicate existing objects.
2- Can you rephrase the question and elaborate ?
3- All the data replication in transit is encrypted between AWS services. You need to ensure the encryption of files at rest by choosing the right encryption option.
4- As both source and destination will be on the same region, this replication will occur on AWS internal network. To observe the throughput , I'd propose de run a test as network is just one of the elements in the equation and you need to consider end to end transfer.
5- The cost elements will include : the number of GET at the source, the number of PUT at the destination(so you need to know the number of objects). As you are in the same region, there is no network charge for the transfer. for more details : https://aws.amazon.com/s3/pricing/
-
The replication rules are configured in the source bucket as below:
- path = xxxxx/xxxxxxx/Folder A
- path = xxxxx/xxxxxxx/Folder B
- path = xxxxx/xxxxx/Folder C
- path = xxxxx/xxxxx/Folder D
if I create a batch replication job, will the existing data in all 4 folders be replicated in the destination bucket with same bucket key/path?
- The service quota for Replication transfer rate in the source account is 1 GBPS. Does it have any relation to transfer rate?
-
Are the current replication rules pointing to the same destination bucket that you will use for migration ? In this case, what is the use case to replicate 'again' objects that are replicated to the same bucket ? In general sense, batch replication replicates all existing objects.
Relevant content
- Accepted Answerasked 10 months ago
AWS OFFICIALUpdated 7 months ago- AWS OFFICIALUpdated 9 months ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated a year ago
The replication rules are configured in the source bucket as below:
if I create a batch replication job, will the existing data in all 4 folders be replicated in the destination bucket with same bucket key/path?
Lastly just FYI, Replication rule basically creates the manifest, it's the replication job which copies the data. Since you are trying to do it for existing data, so first replication rule will ask you whether you want to do it for existing data or not, and as you'd choose yes, it'd create batch job to replicate the data for the prefix which you 'd have given in first replication rule. Catch is, after first rule, it won't ask whether you want to do it for existing data or not. So, I'd suggest you to create manifest for each location first and then create replication job using those manifest files.
Hope it helps, if you have any additional questions, feel free to post here otherwise please click "Accept Answer" and upvote it.