How do I migrate my Storage resources to another region?

Content level: Foundational Context and purpose: This article provides general guidance on migrating Storage resources from one region to another. Delete this section including the line break before publishing the Re:post

---

If you are migrating data from the Middle East (UAE) Region (me-central-1), then you might experience increased error rates as we continue making progress with restoration efforts. For additional information about recovery efforts and service updates that impact your AWS accounts, see the AWS Personal Health Dashboard. For assistance with this event, contact AWS Support through the AWS Management Console or the AWS Support Center.

This is part of a series of articles that provide general guidance on migrating resources form one Region to another. It covers the following services:

• Amazon FSx for NetApp ONTAP
• Amazon FSx for Windows File Server
• Amazon Simple Storage Service (Amazon S3)
• AWS Storage Gateway
• AWS Backup

For general guidance and a full list of domain and service-specific migration guides, see How do I migrate my resources to another region?

For other domains, see the following resources:

• How do I migrate my Security, Identity and Compliance resources to another region?
• How do I migrate my Compute and Container resources to another region?
• How do I copy an EBS Snapshot to another region using coldsnap and the EBS Direct APIs?
• How do I migrate my Database resources to another region?
• How do I migrate my Database resources to another region using a logical dump?
• How do I migrate my Networking and Content Delivery resources to another region?
• How do I migrate my Application Integration resources to another region?

Amazon FSx for NetApp ONTAP

Key Considerations and Preparation

• Backups for FSx for ONTAP file systems do not support copying to additional regions
• SnapMirror requires network connectivity between source and destination file systems, and ONTAP CLI access must be available on both
• Ensure the destination file system has sufficient storage before performing a restore and SSD storage utilization should not exceed 80%
• Before starting any migration, review Monitoring SSD storage utilization to track capacity during the transfer

Using NetApp SnapMirror

SnapMirror is a one-way ONTAP-native block-level replication. This is the recommended approach when the source file system is available and both systems have ONTAP CLI access.

1. Create a file system in the destination region
2. Create the destination volume on the new file system
3. Create the cluster and SVM peering relationship between source and destination
4. Create SnapMirror relationships and begin data transfer from source to destination
5. Repeat the SnapMirror relationship for each volume, cutoverand confirm access to data in the destination

Using AWS DataSync

DataSync copies data from one file system to another. It is simpler than SnapMirror but has additional costs while the transfer is in progress.

1. Create a file system in the destination region
2. Create a volume on the destination file system
3. Create the DataSync locations, task and transfer files to begin migration

Recovering from FSx Backups

This approach is used only if the source file system is completely unavailable. Recover to a working availability zone first, then transfer to a new region using SnapMirror or DataSync.

1. Create a file system in a working availability zone
2. Restore the backup to a new volume on the working file system
3. Once data access is restored, follow the SnapMirror or DataSync approaches above to migrate to a secondary region

---

Amazon FSx for Windows File Server

Key Considerations and Preparation

• Restoring a backup creates a new file system
• DNS aliases must be recreated on the destination file system
• Using DataSync or Robocopy requires connectivity between the source and destination file systems
• Using DataSync or Robocopy will require re-creating the shares on the destination file system
• For DataSync it is recommended the service account is a member of the file system administrators group and has full access to all files on all shares
• HDD file systems will have slower restore times when using DataSync or Robocopy

Using FSx Backups

This is the simplest approach for moving data across regions. FSx for Windows File Server supports copying backups to other regions, unlike FSx for NetApp ONTAP.

1. Copy the backups to the desired region
2. Restore the backup to a new file system in the destination region

Using AWS DataSync

DataSync is an alternative method that has additional costs and allows selective data-copying.

1. Create a file system in the destination region
2. Create SMB shares in the destination to match the source
3. Register locations, create the task and begin the data transfer using DataSync

Using Robocopy

Robocopy is a manual method equivalent to copy-and-paste. This has lower costs than DataSync but requires additional systems administration.

1. Create a file system in the destination region
2. Create SMB shares in the destination to match the source
3. Create the Robocopy operation to copy files from source to destination

Alternative Guides

For a guided approach on using DataSync, refer to Migration to Amazon FSx for Windows File Server: cost optimization strategies and automation.

---

Amazon Simple Storage Service (Amazon S3)

Migrate Amazon S3 Data to Another Region

• KMS keys are regional resources - you cannot use a key from one region to decrypt or re-encrypt objects in another region
• When using SSE-KMS encryption, the KMS key must be in the same region as the S3 bucket
• Versioning must be enabled on both source and destination buckets for S3 replication
• Ensure the IAM principal has kms:Decrypt permissions on the source bucket, kms:GenerateDataKey permissions on the destination bucket, and if using multipart upload, kms:Decrypt permissions on the destination bucket
• Consider enabling S3 Bucket Keys on the destination bucket to reduce KMS costs
• For KMS-encrypted objects using S3 Replication, configure the destination KMS key in the replication configuration following Replicating objects created with server-side encryption

Replicate Data Using S3 Batch Replication

Use this workflow to replicate transactional data for critical workloads to a secondary region.

A script is available that automates replication setup including versioning, IAM role creation, replication configuration, and batch job initiation. It works for any source region by passing the --source-region argument.

1. Download the replication script from the AWS Health Tools S3 replication script repository
2. Run the script with your source and destination bucket parameters to configure replication and initiate the S3 Batch replication job
3. For KMS-encrypted objects, specify the destination region KMS key using the --sse-kms-key-id parameter or configure the ReplicaKmsKeyID in the replication configuration

Copy Data Using AWS DataSync

Use this workflow to copy S3 objects between regions using the AWS Datasync.

We recommend using enhanced mode for optimal performance.

AWS DataSync can be used as an alternative option if you do not need to retain the object creation time, object version IDs, or copy all versions of the objects.

1. Create an S3 bucket in the destination region
2. Register DataSync S3 locations for source and destination buckets
3. Create the Datasync task and begin the data transfer (use enhanced mode for optimal performance)
4. Monitor your data transfer for any failed operations

Copy Data Using AWS CLI

Use this workflow to copy S3 objects directly between regions using the AWS CLI.

1. Use the aws s3 cp command to copy single objects, or add the --recursive flag to copy entire buckets
2. For large-scale transfers, adjust max-concurrent-requests and multipart-chunksize parameters in your S3 CLI configuration for improved performance
3. For significantly better performance, enable the AWS Common Runtime (CRT) S3 client by setting the AWS_USE_CRT=true environment variable or adding use_crt_http_client = true to your AWS CLI config profile (AWS CLI v2 already includes CRT support)
4. For KMS-encrypted destinations, specify the KMS key using the --sse-kms-key-id parameter with the key ARN, or objects will be encrypted using the default KMS key applied to the bucket
5. Use the aws s3 sync command to sync directories and S3 prefixes between buckets for any failed operations.

Restore from AWS Backup

Use this workflow if you have existing S3 backups managed by AWS Backup.

1. Create a backup vault in the destination region
2. Copy the existing backup into another region
3. Restore the backup by resource type to complete the migration

Upload Data from External Sources

Use this workflow if you have backup data available from an external source outside of AWS.

1. Create a new S3 bucket in the target region
2. Upload data using the aws s3 sync CLI command or configure AWS DataSync with an Amazon S3 location

---

AWS Storage Gateway

The migration process differs for each gateway type: S3 File Gateway, Tape Gateway, and Volume Gateway. Refer to gateway specific section below for instructions for migrating your AWS Storage Gateway deployment from one AWS region to another.

S3 File Gateway

The process entails copying your S3 data to the designated region and subsequently recreating your file shares with identical configurations on a gateway within the destination region.

Key Considerations and Preparation

• AWS credentials with permissions for S3, Storage Gateway, KMS, and IAM in both regions.
• Plan sufficient time for data replication (depending on data volume and chosen method) from the source S3 bucket to the destination S3 bucket.
• S3 replication requires versioning, which increases storage costs. DataSync incurs per-GB transfer fees. Both methods incur cross-region data transfer charges.
• Initial access to files in the new region may be slower until the cache is populated.

Migration Steps

1. Select one of the methods outlined in Amazon Simple Storage Service (Amazon S3) section above to transfer your S3 data to the designated region. We strongly recommend utilizing AWS DataSync for this purpose, as it does not necessitate versioning on either bucket. Proceed with this process for each S3 bucket associated with file shares on the source gateway.
2. Create a new S3 File Gateway in the destination region with the same settings and configuration as the source gateway. If necessary, refer to the configuration settings of the source gateway using the “View resource in print-friendly format” feature in the AWS Console.
3. Create file shares (Create an NFS file share, Create an SMB file share) on the new gateway, ensuring the same settings and configuration as those on the old gateway. If required, consult the configuration settings of the file share on the source gateway via “View resource in print-friendly format” in the AWS Console.
4. Confirm that your new gateway is working correctly, then remap/cut-over your NFS/SMB clients from the old file shares to the new file shares aligning with your specific operational requirements..
5. After confirming that your new gateway is working correctly, delete the old gateway from the Storage Gateway console. Before you delete an S3 File Gateway, verify that the CachePercentDirty metric on the Monitoring tab for old gateway is 0. If you delete a gateway while it is in use (CachePercentDirty > 0), data loss can occur.
6. To mitigate potential conflicts arising from concurrent writes, it is advisable to terminate S3 replication or DataSync tasks prior to initiating data writes to file shares on the newly established gateway.

Tape Gateway

To replicate your virtual tapes to another AWS region for disaster recovery, the process entails retrieving the archived tape from the source region and copying the data through a gateway in the destination region.

Key Considerations and Preparation

• AWS credentials with permissions for Storage Gateway and KMS in both regions.
• Plan sufficient time for retrieving archived tapes.
• Tape retrieval incurs retrieval and data transfer fees.
• Retrieved virtual tapes are read-only and cannot be used for writing new backups.
• Initial access to virtual tapes in the new region may be slower until the cache is populated.

Migration Steps

1. Setup a Tape Gateway in the destination region.
2. Connect your backup server to the new gateway and add it as a tape library in your on-premises backup software.
3. Create virtual tapes on the destination gateway to store the data copied from the source tapes.
4. If you have archived tapes, retrieve them to the source gateway first.
5. Use your on-premises backup software to copy tapes from the source to the destination virtual tape library (Tape Gateways).

Volume Gateway

To replicate your volumes hosted on a Volume Gateway to another AWS region, create EBS snapshots of your volumes and copy them to the destination region. Once the EBS snapshots are available in the destination region, you can access them using a new Volume Gateway in that region.

Key Considerations and Preparation

• AWS credentials with permissions for Storage Gateway and KMS in both regions
• Plan sufficient time for data replication for EBS snapshot copy to finish
• Initial read access to volumes in the new region may be slower until cache is populated
• The instructions provided are specific to Cached Volume Gateway.

Migration Steps

1. Invoke the Storage Gateway CreateSnapshot API for each volume on the source gateway.
2. Wait for the snapshots to reach the completed state in the EBS console. See Describing snapshots for details.
3. Invoke the EBS CopySnapshot API for each snapshot, specifying the destination region in the request.
4. To use the volume in the destination region, create a new Volume Gateway in that region, then invoke the Storage Gateway CreateCachediSCSIVolume API, providing the ID of the copied snapshot, to recreate your volume on the new gateway.

---

AWS Backup

Copy Existing Backups (One-Time Migration)

If you already have an existing backup, you can copy a recovery-point into another region and later restore.

1. Create a backup vault in the destination region
2. Copy the existing backup into another region
3. Restore the backup by resource type to complete the migration

Create an On-Demand Backup

If you're using AWS Backup for the first time and don't have any existing backup, you can create an on-demand backup. An on-demand backup begins to back up your resource immediately.

1. Create a backup vault in the destination region
2. Create an on-demand backup of your resource
3. Once the on-demand backup completes, copy the recovery-point into the new region
4. Restore the backup by resource type to complete the migration

---

In collaboration with Vania Toma and Nymus Booysen