Help with copying s3 bucket to another location missing objects


Hello All, Today I was trying to copy a directory from one location to another, and was using the following command to execute my copy.

aws s3 s3://bucketname/directory/ s3://bucketname/directory/subdirectory --recursive

The copy took overnight to complete because it was 16.4TB in size, but when I got into work the next day, it was done, or at least it had completed.

But when I do a compare between the two locations I get the following

bucketname/directory/ 103,690 objects - 16.4TB

bucketname/directory/subdirectory/ 103,650 - 16.4TB

So there is a 40 object difference between the source location and the destination location.

I tried using the following command to copy over the files that were missing

aws s3 sync s3://bucketname/directory/ s3://bucket/directory/subdirectory/

which returned no results. It sat for a while maybe like 2 minutes or so, and then just returned to the next line.

I am at my wits end trying to copy of the missing objects, and my boss thinks that I lost the data, so I need to figure out a way to get the difference between the source and destination copied over.

If anyone could help me with this, I would REALY appreciate it. I am a newbie with AWS, so I may not understand everything that I am told, but I will try anything to get this resolved. I am doing all the commands through an EC2 instance that I am ssh into, and then use AWS CLI commands.

Thanks to anyone who might be able to help me.

Take care,

-Tired & Frustrated :)

1 Answer

I can't answer for the discrepancy between the number of objects. Rather than using an EC2 instance to synchronize the buckets, please check out the resources below.

  1. Replicating existing objects between S3 buckets
  2. Amazon S3 Batch Replication
  3. AWS DataSync
profile pictureAWS
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions