S3DistCp on a bucket with Requester Pays


Is there a way to use s3-dist-cp to copy files from a bucket that uses Requestor payments?

asked 4 months ago291 views
2 Answers

Please find the example for requester pay model s3-dist-cp --src=s3://[requester-pays-bucket]/[path] --dest=s3://[destination-bucket]/[path] You need to replace request-paybucketname and destinationbucket as above commands.

Since you are the requester, you will be billed for data transfer and request costs. Ensure that you are aware of the pricing implications.

profile picture
answered 4 months ago


Because requesters are charged for downloading data from Requester Pays buckets, the requests must contain a special parameter, x-amz-request-payer, which confirms that the requester knows that they will be charged for the download. To do this, if you are utilizing EMRFS, you can set fs.s3.useRequesterPaysHeader to true in /usr/share/aws/emr/emrfs/conf/emrfs-site.xml. Or, setting fs.s3a.requester.pays.enabled: true with S3A connector. You can also refer to this article for another example.

I hope this helps.

answered 3 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions