How can I use a Lambda function to copy files from one Amazon S3 bucket to another?

4 minute read
0

I want to use an AWS Lambda function to copy files from an Amazon Simple Storage Service (Amazon S3) bucket to another bucket.

Resolution

Follow these steps to create a Lambda function that copies files from a source Amazon S3 bucket to a destination S3 bucket.

Create the source and destination Amazon S3 buckets

Note: If you already created the source and destination S3 buckets, then skip this step.

Complete the following steps:

  1. Open the Amazon S3 console.
  2. Choose Create bucket.
  3. For Bucket name, enter a name for the source bucket.
  4. From the AWS Region dropdown list, choose your AWS Region.
  5. Choose Create bucket.
  6. Repeat steps 1-4 for your destination bucket.

For more information, see Creating a bucket.

Create the Lambda function

Complete the following steps:

  1. Open the Functions page in the Lambda console.

  2. Choose Create function, and then, choose Author from scratch.

  3. For Function name, enter a name for your function.

  4. From the Runtime dropdown list, choose Python 3.11.

  5. Expand Change default execution role, and then choose Create a new role with basic permissions.

  6. Choose Create function.

  7. Choose the Code tab, and then paste the following Python code:

    import boto3import botocore
    import json
    import os
    import logging
    logger = logging.getLogger()
    logger.setLevel(logging.INFO)
    
    s3 = boto3.resource('s3')
    
    def lambda_handler(event, context):
        logger.info("New files uploaded to the source bucket.")
            
        key = event['Records'][0]['s3']['object']['key']
            
        source_bucket = event['Records'][0]['s3']['bucket']['name']
        destination_bucket = os.environ['destination_bucket']
        
        source = {'Bucket': source_bucket, 'Key': key}
            
        try:
            response = s3.meta.client.copy(source, destination_bucket, key)
            logger.info("File copied to the destination bucket successfully!")
            
        except botocore.exceptions.ClientError as error:
            logger.error("There was an error copying the file to the destination bucket")
            print('Error Message: {}'.format(error))
            
        except botocore.exceptions.ParamValidationError as error:
            logger.error("Missing required parameters while calling the API.")
            print('Error Message: {}'.format(error))

    Note: Find the source_bucket name from the event object that the Lambda function receives. You can store the destination_bucket name as an environment variable.

  8. Choose Deploy.

Lambda creates an execution role that grants the function permission to upload logs to Amazon CloudWatch. For more information, see Create a Lambda function with the console.

Create an Amazon S3 trigger for the Lambda function

Complete the following steps:

  1. Open the Functions page in the Lambda console.
  2. In Functions, choose the Lambda function.
  3. In Function overview, choose Add trigger.
  4. From the Trigger configuration dropdown list, choose S3.
  5. In Bucket, enter the name of your source bucket.
  6. From the Event type dropdown list, choose All object create events.
    Note: Configure this step to use either the POST, PUT, or Multipart Upload event type to upload the object.
  7. Select the I acknowledge that using the same S3 bucket for both input and output is not recommended agreement, and then choose Add.

For more information, see Tutorial: Using an Amazon S3 trigger to invoke a Lambda function.

Provide IAM permissions for the Lambda function's execution role

To copy files to the destination S3 bucket, add AWS Identity and Access Management (IAM) permissions for the Lambda function's execution role. Use a policy similar to the following resource-based policy:

{  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "putObject",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject"
      ],
      "Resource": [
        "arn:aws:s3:::destination-s3-bucket/*"
      ]
    },
    {
      "Sid": "getObject",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject"
      ],
      "Resource": [
        "arn:aws:s3:::source-s3-bucket/*"
      ]
    }
  ]
}

Note:

  • Replace destination-s3-bucket with your S3 destination bucket and source-s3-bucket with your S3 source bucket.
  • Replace /* at the end of the resource ARN with the required prefix value for your environment to limit permissions.
  • It's a best practice to grant least privilege permissions to perform a task.
  • To copy objects that have object tags, the execution role must have s3:GetObjectTagging and s3:PutObjectTagging permissions. You must have s3:GetObjectTagging permission for the source object in the source bucket and s3:PutObjectTagging permission for objects in the destination bucket.

For more information, see Granting function access to AWS services.

Related information

How do I troubleshoot 403 Access Denied errors from Amazon S3?

How do I allow my Lambda execution role to access my Amazon S3 bucket?

AWS OFFICIAL
AWS OFFICIALUpdated 5 months ago
4 Comments

I would like to add to this article that if you are copying object with tags you would also need the ["s3:GetObjectTagging", "s3:PutObjectTagging"] actions to the Lambda function role. This is needed because the Boto3 copy_object method "TaggingDirective" argument is set to “COPY” by default.

replied 6 months ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
MODERATOR
replied 6 months ago

In the sample python code provided above, there's a typo on the first line:

import boto3import botocore

...should be...

import boto3

import botocore

replied 4 months ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
MODERATOR
replied 4 months ago