How can I migrate my Amazon DynamoDB tables from one AWS account to another?

Short description

Migrate your DynamoDB table to a different AWS account with one of these methods that suit your use case:

• AWS Backup
• DynamoDB import and export to Amazon Simple Storage Service (Amazon S3)
• Amazon S3 and AWS Glue
• Amazon EMR

Resolution

AWS Backup

You can use AWS Backup to create cross-account DynamoDB backups. For more information, see Creating backup copies across AWS accounts and AWS Backup demo: Cross-account & cross-Region backup.

In the target account, do the following:

1. Create an AWS Backup vault in the target account in the Region where your DynamoDB is present.
2. When you create the vault, use the AWS Key Management Service (AWS KMS) key that you already configured. This is the key that you already shared with the source account in the same organization.
3. After you create the vault, add an AWS Identity and Access Management (IAM) policy to the vault. You can do by selecting the option Allow access to a backup vault from organization. This allows other accounts within the same organization to copy into the vault.

In the source account, do the following:

1. In the source account where your DynamoDB is present, create an AWS Backup vault in the Region where your table data needs to be migrated.
2. When you create the vault, use the AWS KMS key that you already configured. This is the key that you shared with other accounts in the organization.
3. Add an IAM policy to the vault that allows other account in the organization to copy into the vault. You can do by selecting the option You can do by selecting the option Allow access to a backup vault from organization.
4. Create a backup plan to generate backups of the DynamoDB tables in the source account to the target account.
5. For Backup vault, be sure to choose the vault that you created in the source account.
6. Then, select the Copy to another account's vault option.
7. For Assign resources, be sure to include the resources that you must back up. You can choose Include specific resource types.
8. For Select specific resource types, select DynamoDB. You can choose all tables or only those tables that you must back up.

In the target account, do the following:

1. In the target account, navigate to the vault that you created.
   You can see that the Recovery points are the same as in the source account.
2. You can restore your DynamoDB table in the target account.

Note: For this approach, the source and target accounts must be in the same organization.

DynamoDB import and export to Amazon S3

1. Migrate the DynamoDB table data by exporting the table to an Amazon S3 bucket in the target account. Be sure that DynamoDB has s3:ListBucket permissions to this S3 bucket. Be sure that the S3 bucket doesn't have any access control lists that deny access to the exported data.
2. After the export is completed, import the data from the S3 bucket into a new table in the target account. For more information, see DynamoDB data import from Amazon S3: how it works.

Note: Exporting the table doesn't consume any read capacity on the table and has no impact on table performance or availability. Also, importing the table doesn't consume any writer capacity. Therefore, you don't need additional capacity during the import process.

Amazon S3 and AWS Glue

You can migrate your DynamoDB table to a different AWS account using an S3 bucket and an AWS Glue job.

1.    You can perform the initial migration of the DynamoDB table by exporting the tables to an Amazon S3 bucket in the other account.

When you export your tables from Account A to an S3 bucket in Account B, the objects are still owned by Account A. The AWS Identify Access Management (IAM) users in Account B can't access the objects by default. The export function doesn't write data with the access control list (ACL) bucket-owner-full-control. As a workaround to this object ownership issue, include the PutObjectAcl permission on all exported objects after the export is complete. This workaround grants access to all exported objects for the bucket owners in Account B. For more information, see Why can't I access an object that was uploaded to my Amazon S3 bucket by another AWS account?

2.    Use a Glue job to read the files from the S3 bucket and write them to the target DynamoDB table.

After you export the data to an S3 bucket in the target account as mentioned in step 1, do the following in the target account:

Run an AWS Glue crawler on the data in Amazon S3. The crawler infers the schema and creates an AWS Glue Data Catalog table with that schema definition.

Use the AWS Glue Studio to create an ETL job. After you specify a source, transformation, and target, AWS Glue Studio automatically generates the PySpark code based on these inputs. For this job, specify the AWS Glue Data Catalog table as the source and ApplyMApplyMapping as the transformation. Don't specify a target. AWS Glue Studio generates the PySpark code to create a DynamicFrame from S3.

Be sure that the key name and data type mapping of the code generated by AWS Glue Studio are correct. If the mappings aren't correct, then modify the code, and correct the mappings. Because you didn't specify the target when creating the AWS Glue job, add a sink operation similar to the following in this example. Adding this operation allows the job to write directly to the target DynamoDB table:

glueContext.write_dynamic_frame_from_options ( frame = Mapped, connection_type = "dynamodb", connection_options = { "dynamodb.region": "", "dynamodb.output.tableName": "", "dynamodb.throughput.write.percent": "1.0" } )

To load the data to the target table, run the job from AWS Glue Studio or the Jobs page on the AWS Glue console.

3.    After you export the tables to the S3 bucket, use DynamoDB Streams and AWS Lambda to migrate data insertions and updates from the source table to the destination table in another account. For more information, see Cross-account replication with Amazon DynamoDB.

AWS Glue

AWS Glue ETL jobs support both reading data from and writing data into another account's DynamoDB table. Use the dynamodb.sts.roleArn parameter to assume a cross-account role in the job script. Assuming this role allows you to get the temporary credentials that must be used for cross-account access to DynamoDB. For more information, see Cross-account cross-Region access to DynamoDB tables and How to export an Amazon DynamoDB table to Amazon S3 using AWS Step Functions and AWS Glue.

Amazon EMR

When you use Amazon EMR to migrate DynamoDB tables, use one of the following options depending on your use case:

• If you can afford downtime during the migration, then stop write operations to the source table. This is to make sure that the target table is in sync with the source table.
• If you can't afford downtime, then you must store all transactions that happen during the migration in a staging table. After the original table is migrated to the other AWS account, push the new transactions from the staging table to the target table.

Note: The time required to migrate tables with Amazon EMR can vary significantly. The variance depends upon the DynamoDB table's provisioned throughput network performance and the amount of data stored in the table.

To migrate a DynamoDB table using Amazon EMR, do the following:

1. Launch EMR clusters in both the source and destination accounts. In the Software configuration section, be sure that you choose an option that includes Apache Hive. Note: It's a security best practice to launch Amazon EMR clusters into private subnets. The private subnets must have an Amazon S3 VPC endpoint and a route to DynamoDB. For more information, see Private subnets. If the clusters must access the internet, then use a NAT gateway that resides in a public subnet. For more information, see VPC with public and private subnets (NAT).
2. Be sure that the EMR_EC2_DefaultRole IAM roles in both accounts have permission to write to the S3 bucket in the destination account. For more information, see Configure IAM service roles for Amazon EMR permissions to AWS services and resources.
3. In the source account, connect to the master node using SSH.
4. In the source account, use Hive commands to export the DynamoDB table data to the S3 bucket in the destination account.
5. In the destination account, import the Amazon S3 data to the new DynamoDB table.
6. If you're using a staging table to capture writes that happened during the migration, repeat steps 4 and 5 on the staging table.

Related information

Copying data between DynamoDB and Amazon S3