Copy files from one S3 bucket to another S3 bucket

Hi bharath

This error indicates that the IAM role associated with your Glue job lacks the necessary permissions to copy objects from the source bucket to the target bucket in the other account.

Verify IAM Role Permissions:

Ensure the Glue job's IAM role has the s3:GetObject permission for the source bucket and the s3:PutObject permission for the target bucket.

Consider attaching the following managed policies to the IAM role: AmazonS3ReadOnlyAccess for source bucket access AmazonS3FullAccess for target bucket access (if needed for full control) or a custom policy with specific s3:PutObject actions.

IAM Policy Simulator: https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_testing-policies.html

S3 Bucket Policies: https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucket-policies.html

IAM Roles for Cross-Account Access: https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html

Hi,

Also be sure that your properly set up the cross-account authorizations for your bucket.

See https://repost.aws/knowledge-center/cross-account-access-s3 for all details on how to do it.

Best,

Didier

Hello Bharat,

Transfer the large files best option is Assume Role in Target Account:

Target Account Setup:

• Create an IAM role in the target account with permissions to write objects to the target bucket.
• Attach a policy that allows "s3:PutObject" access to the target bucket.
• Configure a trust relationship for the role to allow your Glue job's role (source account) to assume it.

Grant Glue job role access to target bucket's IAM role with write permissions. https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html

Assuming roles in other accounts isn't going to help. You need two things:

1. The IAM role that your Glue job is running under (without assuming any roles elsewhere) needs to have the s3:GetObject and possibly s3:GetObjectVersion actions for objects in the source bucket. It may also need the s3:GetBucketLocation and s3:ListBucket permissions to the source bucket. Since the IAM role is in the same account as the source bucket, it'll suffice to grant these permissions in the permissions policy attached to the IAM role, without having to touch the bucket policy of the source bucket.

2. The same IAM role that the Glue job is running under (and not another role assumed from the other account) needs to have the s3:PutObject and possibly s3:PutObjectAcl permissions to objects in the destination bucket. You must grant these permissions both in the permissions policy attached to the IAM role in the source account and in the bucket policy of the S3 bucket in the destination account.

If you're using SSE-KMS encryption for either bucket, the IAM role additionally needs to be granted the kms:Decrypt and kms:GenerateDataKey permissions to the KMS keys. If you're using the default SSE-S3 encryption option for both buckets, KMS won't be relevant.

The important general point is that it's the principal that was used to call CopyObject (in this case, the IAM role of your Glue job) that will access both the source and destination buckets. There's no benefit to assuming a role in the destination account, which would then need the cross-account access to the source bucket, as opposed to simply using the initial execution role of your Glue job, located in the source account, and being granted cross-account access to the destination bucket. Creating and assuming an additional role would only add complexity with zero added value.

I am not able to upload the file to target using boto3 (https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/copy.html), But I am able to upload the file to target bucket using multipart upload.

It seems boto copy can be used if the source and target are in account and similar permissions. For Ex : s3_client.copy(**args). Here we don't have any parameter to use 2 different clients.