Migrate on-prem data( files) to S3 using Snowball and data sync using DataSync

0

Customer wants to migrate 900 TB of unstructured data( file data) and this data continues to change/update. We are thinking to use snowball for the data migration and datasync to sync the incremental data after the migration. Can we use DataSync to transfer any incremental changes after the migration (something like DMS used to migrate database CDC) ? what are the other options to sync the incremental changes ?

AWS
已提问 2 年前582 查看次数
1 回答
1

There are a few different options in this case and I definitely recommend reaching out to your AWS account contacts so they can offer more assistance.

Firstly the use of Snowball will depend on the speed of your network links, the rate of data change, migration cutover window and data security requirements. eg 900TB can be transferred over a multi tunnel vpn using s3 multipart upload if your internet links are good, security policies allow and the rate of change isn't high. The aws s3 sync command can subsequently perform sync tasks. If you have to use Snowball standard s3 commands can be used to copy the device to snowball and subsequently sync (depending on number this can take a lot of time) but you do not have to use DataSync

DataSync as you suggested is something to consider and can also be used with Snow appliances. If you need to run the sync tasks on a defined schedule, control the bandwidth and task duration it should definitely be considered over standard s3 cli commands as offers more options and optimisations....and yes it handles incremental (only transfers changed files). See this blog

The other main option to consider is the Storage Gateway appliances. If you are going to be operating in a hybrid mode with data both onpremise and in AWS for an extended period of time then this becomes an attractive option

AWS
专家
Peter_G
已回答 2 年前
profile picture
专家
已审核 24 天前

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。

回答问题的准则