Why aren't my Amazon S3 objects replicating when I set up replication between my buckets?

8 minute read
0

I set up cross-Region replication (CRR) or same-Region replication (SRR) between my Amazon Simple Storage Service (Amazon S3) buckets. However, objects aren't replicating to the destination bucket.

Resolution

To troubleshoot S3 objects that aren't replicating to the destination bucket, check the different types of permissions for your bucket. Also, check the public access settings and bucket ownership settings.

Tip: Upload an object to the source bucket to test the replication after each configuration change. It's a best practice to make one configuration change at a time to identify any replication setup issues.

After you resolve the issues causing replication to fail, there might be objects in the source bucket that weren't replicated. By default, S3 replication doesn't replicate existing objects or objects with a replication status of FAILED or REPLICA. To check the replication status for objects that didn't replicate, see Retrieve list of S3 objects that failed replication. Use S3 batch replication to replicate these objects.

Minimum Amazon S3 permissions

Confirm that the AWS Identity Access Management (IAM) role that you used in the replication rule has the correct permissions. If the source and destination buckets are in different accounts, confirm that the destination account's bucket policy grants sufficient permissions to the replication role.

The following example shows an IAM policy with the minimum permissions that are required for replication. Based on the replication rule options (example: encryption with SSE-KMS), you might need to grant additional permissions.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetReplicationConfiguration",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::SourceBucket"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObjectVersionForReplication",
        "s3:GetObjectVersionAcl",
        "s3:GetObjectVersionTagging"
      ],
      "Resource": [
        "arn:aws:s3:::SourceBucket/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:ReplicateObject",
        "s3:ReplicateDelete",
        "s3:ReplicateTags"
      ],
      "Resource": "arn:aws:s3:::DestinationBucket/*"
    }
  ]
}

Note: Replace SourceBucket and DestinationBucket with the names of your S3 buckets.

The IAM role must have a trust policy that allows Amazon S3 to assume the role to replicate objects.

Example:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

If the destination bucket is in another account, then the destination bucket policy must grant the following permissions:

{
  "Version": "2012-10-17",
  "Id": "PolicyForDestinationBucket",
  "Statement": [
    {
      "Sid": "Permissions on objects",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::SourceBucket-account-ID:role/service-role/source-account-IAM-role"
      },
      "Action": [
        "s3:ReplicateTags",
        "s3:ReplicateDelete",
        "s3:ReplicateObject"
      ],
      "Resource": "arn:aws:s3:::DestinationBucket/*"
    }
  ]
}

Note: Replace arn:aws:iam::SourceBucket-account-ID:role/service-role/source-account-IAM-role with the ARN of your replication role.

Additional Amazon S3 permissions

If the replication rule is set to Change object ownership to the destination bucket owner, then the IAM role must have the s3:ObjectOwnerOverrideToBucketOwner permissions. This permission is placed on the S3 object resource.

Example:

{
    "Effect":"Allow",
         "Action":[
       "s3:ObjectOwnerOverrideToBucketOwner"
    ],
    "Resource":"arn:aws:s3:::DestinationBucket/*"
}

The destination account must also grant the s3:ObjectOwnerOverrideToBucketOwner permission through the bucket policy:

{
    "Sid":"1",
    "Effect":"Allow",
    "Principal":{"AWS":"arn:aws:iam::SourceBucket-account-ID:role/service-role/source-account-IAM-role"},
    "Action":["s3:ObjectOwnerOverrideToBucketOwner"],
    "Resource":"arn:aws:s3:::DestinationBucket/*"
}

Note: If the destination bucket's object ownership settings include Bucket owner enforced, then you don't need Change object ownership to the destination bucket owner in the replication rule. This change occurs by default.

If the replication rule has delete marker replication activated, then the IAM role must have the s3:ReplicateDelete permissions. If the destination bucket is in another account, then the destination bucket owner must also grant this permission through the bucket policy. For example:

{
    "Effect":"Allow",
    "Action":["s3:ReplicateDelete"],
    "Resource":"arn:aws:s3:::DestinationBucket/*"
}

Note: Replace DestinationBucket with the name of your S3 bucket.

After you add the additional S3 permissions to the IAM role, the same permissions must be granted through the bucket policy on the destination bucket:

{
  "Version": "2012-10-17",
  "Id": "PolicyForDestinationBucket",
  "Statement": [
    {
      "Sid": "Stmt1644945277847",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::SourceBucket-account-ID:role/service-role/source-account-IAM-role"
      },
      "Action": [
        "s3:ReplicateObject",
        "s3:ReplicateTags",
        "s3:ObjectOwnerOverrideToBucketOwner",
        "s3:ReplicateDelete"
      ],
      "Resource": "arn:aws:s3:::DestinationBucket/*"
    }
  ]
}

AWS KMS permissions

If a bucket's source objects are encrypted with an AWS KMS key, then the replication rule must be configured to include AWS KMS-encrypted objects.

To include objects encrypted with AWS KMS, do the following:

1.    Open the Amazon S3 console.

2.    Choose the S3 bucket that contains the source objects.

3.    On the Management tab, select a replication rule.

5.    Choose Edit.

6.    Under Encryption, choose Replicate objects encrypted with AWS KMS.

7.    Under AWS KMS key for encrypting destination objects, select an AWS KMS key. The default option is to use the AWS KMS key (aws/S3).

Examples: Example policies – Using SSE-S3 and SSE-KMS with replication

Important: If the destination bucket is in a different AWS account, then specify an AWS KMS customer managed key that's owned by the destination account. Don't use the default aws/S3 key. This encrypts the objects with the AWS managed key that's owned by the source account and can't be shared with another account. As a result, the destination account can't access the objects in the destination bucket.

AWS KMS additional permissions for cross-account scenarios

To use an AWS KMS key that belongs to the destination account to encrypt the destination objects, the destination account must grant the replication role in the AWS KMS key policy:

{
    "Sid": "AllowS3ReplicationSourceRoleToUseTheKey",
    "Effect": "Allow",
    "Principal": {"AWS": "arn:aws:iam::SourceBucket-account-ID:role/service-role/source-account-IAM-role"},
    "Action": ["kms:GenerateDataKey", "kms:Encrypt"],
    "Resource": "*"
}

Note: Suppose that you use an asterisk (*) for Resource in the AWS KMS key policy. In this case, the policy grants permission for the AWS KMS key only to the replication role. The policy doesn't allow the replication role to elevate its permissions.

Additionally, the source account must add the following minimum permissions to the replication role's IAM policy:

{
    "Effect": "Allow",
    "Action": [
        "kms:Decrypt",
        "kms:GenerateDataKey"
    ],
    "Resource": [
        "SourceKmsKeyArn"
    ]
},
{
    "Effect": "Allow",
    "Action": [
        "kms:GenerateDataKey",
        "kms:Encrypt"
    ],
    "Resource": [
        "DestinationKmsKeyArn"
    ]
}

By default, the AWS KMS key policy grants the root user full permissions to the key. These you can delegate these permissions can to other users in the same account. You can use an IAM policy to grant the replication role permissions to the source KMS key. This is sufficient unless there are deny statements in the source KMS key policy.

Explicit deny and conditional allow statements

If your objects still aren't replicating after you've validated the permissions, then check for any explicit deny statements:

  • Deny statements in the destination bucket policy or AWS KMS key policies that restrict access to the following might cause replication to fail.
    • Specific CIDR ranges
    • Virtual Private Cloud (VPC) endpoints
    • S3 access points
  • Deny statements or permissions boundaries that are attached to the IAM role might cause replication to fail.
  • Deny statements in AWS Organizations service control policies (SCPs) that are attached to either the source or destination accounts might cause replication to fail.

Tip: Before removing any explicit deny statements, confirm the reason for using the deny and determine whether the statement has an impact on data security.

Amazon S3 bucket keys and replication considerations

If either the source or destination KMS keys grant permissions based on the encryption context, then confirm that the S3 Bucket Keys are turned on for the buckets. If the buckets have Bucket Keys turned on, then the encryption context must be for the bucket level resource:

"kms:EncryptionContext:aws:s3:arn": [
     "arn:aws:s3:::SOURCE_BUCKET_NAME"
     ]
"kms:EncryptionContext:aws:s3:arn": [
     "arn:aws:s3:::DESTINATION_BUCKET_NAME"
     ]

Note: Replace SOURCE_BUCKET_NAME and DESTINATION_BUCKET_NAME with the names of your source and destination buckets.    

If Bucket Keys aren't turned on for the source or destination buckets, then the encryption context must be the object level resource:

"kms:EncryptionContext:aws:s3:arn": [
     "arn:aws:s3:::SOURCE_BUCKET_NAME/*"
     ]
"kms:EncryptionContext:aws:s3:arn": [
     "arn:aws:s3:::DESTINATION_BUCKET_NAME/*"
     ]

Note: Replace SOURCE_BUCKET_NAME and DESTINATION_BUCKET_NAME with the names of your source and destination buckets.

Object ACLs and Block Public Access

Check whether the source and destination buckets are using access control lists (ACLs). If the object includes an ACL that allows public access but the destination bucket is using block public access, then replication fails.

Source object ownership

If the objects in the source bucket were uploaded by another AWS account, then the source account might not have permission to those objects. Check the source bucket to see if ACLs are deactivated. If the source bucket has ACLs deactivated, then the source account is the owner of all objects in the bucket. If the source bucket doesn't have ACLs deactivated, then check to see if object ownership is set to Object owner preferred or Bucket owner preferred. If the bucket is set to Bucket owner preferred, then the source bucket objects must have the bucket-owner-full-control ACL for the bucket owner to become the object owner.

The source account can take ownership of all objects in their bucket by deactivating ACLs. Most use cases don't require using ACLs to manage access. It's a best practice to use IAM and bucket policies to manage access to S3 resources. To deactivate ACLs on your S3 bucket, see Controlling ownership of objects and disabling ACLs for your bucket. Be sure to evaluate the current usage of ACLs on your bucket and objects. Your current bucket and IAM policies must grant sufficient permissions so that you can deactivate ACLs without impacting Amazon S3 access.

Replication rule filter

Be sure that you specified the replication rule filter correctly.

If you specify a rule filter with a combination of a key prefix and object tags, S3 performs a logical AND operation to combine these filters. In other words, the rule applies to a subset of objects with a specific key prefix and specific tags.

Related information

Walkthroughs: Examples for configuring replication

AWS OFFICIAL
AWS OFFICIALUpdated 9 months ago
2 Comments

In order to monitor the replication status of your existing objects, configure Amazon S3 Inventory on the source bucket at least 48 hours prior to enabling the replication. For detailed instructions on setting this up, see the user guide on configuring Amazon S3 Inventory. مكاتب مفروشة للايجار مكاتب مجهزة للايجار

You can query S3 Inventory using AWS CLI as described here or by using Athena as shown in this blog post. Replication status provides information on whether the object replication is pending, completed, or failed.

rkk
replied 10 months ago

It is evident that ensuring proper permissions, verifying existing objects, and considering ownership and ACLs are crucial steps in resolving replication issues. Implementing these recommendations will help diagnose and address the replication challenges effectively.

profile picture
Amol_M
replied 8 months ago