Migrate on-prem data( files) to S3 using Snowball and data sync using DataSync

0

Customer wants to migrate 900 TB of unstructured data( file data) and this data continues to change/update. We are thinking to use snowball for the data migration and datasync to sync the incremental data after the migration. Can we use DataSync to transfer any incremental changes after the migration (something like DMS used to migrate database CDC) ? what are the other options to sync the incremental changes ?

AWS
asked 2 years ago545 views
1 Answer
0

There are a few different options in this case and I definitely recommend reaching out to your AWS account contacts so they can offer more assistance.

Firstly the use of Snowball will depend on the speed of your network links, the rate of data change, migration cutover window and data security requirements. eg 900TB can be transferred over a multi tunnel vpn using s3 multipart upload if your internet links are good, security policies allow and the rate of change isn't high. The aws s3 sync command can subsequently perform sync tasks. If you have to use Snowball standard s3 commands can be used to copy the device to snowball and subsequently sync (depending on number this can take a lot of time) but you do not have to use DataSync

DataSync as you suggested is something to consider and can also be used with Snow appliances. If you need to run the sync tasks on a defined schedule, control the bandwidth and task duration it should definitely be considered over standard s3 cli commands as offers more options and optimisations....and yes it handles incremental (only transfers changed files). See this blog

The other main option to consider is the Storage Gateway appliances. If you are going to be operating in a hybrid mode with data both onpremise and in AWS for an extended period of time then this becomes an attractive option

AWS
EXPERT
Peter_G
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions