How do I copy all objects from one Amazon S3 bucket to another bucket?

5 minute read

I want to copy or move all my objects from one Amazon Simple Storage Service (Amazon S3) bucket to another bucket.

Short description

To copy objects from one S3 bucket to another bucket, choose one of the following options:

Run the sync command from the AWS Command Line Interface (AWS CLI)
Use S3 Batch Operations for buckets that contain a lot of objects

The sync command works well for buckets that aren't very large. But it's expensive to run the sync command to copy large buckets with millions of objects, and the copy operation might time out. If you experience timeouts when you copy a bucket, then use Amazon CloudWatch metrics to calculate the size and number of objects in the bucket. If you have millions of objects in your S3 bucket, then it's a best practice to use S3 Batch Operations to copy your objects.

Note: To copy objects from your bucket to a bucket owned by a different AWS account, use AWS Identity and Access Management (IAM) to set up cross-account permissions. Make sure that your IAM role has s3:GetObjectTagging permissions for source objects and s3:PutObjectTagging permissions for destination objects.

To increase sync performance and copy your objects faster, do one or more of the following:

Tune the AWS CLI to use a higher concurrence
Split the process into multiple sync commands based on object name prefixes
Organize your S3 bucket architecture according to best practices

Resolution

Note: If you receive errors when you run AWS CLI commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Copy objects between the S3 buckets

To copy objects from one bucket to another bucket, complete the following steps:

If you archived S3 objects in the Amazon Simple Storage Service Glacier storage class, then restore the objects.
Run the following sync command, but replace example_source_bucket and example_target_bucket with your S3 bucket names:
```
aws s3 sync s3://example_source_bucket s3://example_target_bucket
```
Note: The sync command copies only those objects that aren't already in the destination bucket. To explicitly copy every object, use the cp command instead of the sync command. When you use the sync command on a versioned bucket, only the current version of each object is copied. By default, this behavior preserves object metadata.

(Optional) If you encounter a timeout, then use the CloudWatch get-metric-statistics command to calculate your bucket size and number of objects.

To calculate the size of your bucket, run the following command:

aws cloudwatch get-metric-statistics --namespace AWS/S3 --metric-name BucketSizeBytes --dimensions Name=BucketName,Value=example_source_bucket Name=StorageType,Value=StandardStorage --start-time 2021-05-11T00:00 --end-time 2021-05-11T00:10 --period 3600 --statistics Average --unit Bytes --output json

To calculate the number of objects in your bucket, run the following command:

aws cloudwatch get-metric-statistics --namespace AWS/S3 --metric-name NumberOfObjects --dimensions Name=BucketName,Value=example_source_bucket Name=StorageType,Value=AllStorageTypes --start-time 2021-05-11T00:00 --end-time 2021-05-11T00:10 --period 600 --statistic Average --output json

If your bucket size is large and it contains millions of objects, then don't use the sync command. Instead, use S3 Batch Operations when you copy your bucket.

Important: Access control lists (ACLs) aren't copied from the source bucket to the destination bucket when you run the sync command. If both the source and destination bucket have active ACLs, then the destination object ACLs grant FULL_CONTROL to the account that performed the copy. To troubleshoot issues with the sync command, see Why can't I copy an object between two Amazon S3 buckets?

Verify that the objects were copied

To verify that the objects were copied, complete the following steps:

To verify the contents of the source and target buckets, run the following ls commands:

aws s3 ls --recursive s3://example_source_bucket --summarize > bucket-contents-source.txt

aws s3 ls --recursive s3://example_target_bucket --summarize > bucket-contents-target.txt

Note: The ls command might time out for large buckets. For large buckets, use CloudWatch metrics to calculate the size of the bucket and total number of objects instead of the ls command. However, because Amazon CloudWatch metrics are pulled only once per day, the CloudWatch results might differ from the ls command results.

Use the text files that were output from the previous ls commands to compare objects between the source and target buckets. The output is similar to the following example:

2017-11-20 21:17:39      15362 s3logo.png

  Total Objects: 1        Total Size: 15362

Note: If you have applications or workloads that make API calls to the source bucket, then update those API calls to the target bucket. If you have frequent writes, then you might need to run sync commands to address discrepancies between the source and target buckets.