How do I use custom resources with Amazon S3 buckets in CloudFormation?

Resolution

CloudFormation templates use AWS Lambda backed custom resources. When you use a CloudFormation template for custom resources with an S3 bucket, you can take the following actions:

• Use the template to create folders in S3 buckets.
• Use the template to copy, upload, or synchronize content across two buckets after you create an S3 bucket.
• Modify the template with your own code.

Note: In the following resolution, Amazon S3 deletes all the S3 bucket content when you delete the CloudFormation stack. To modify this behavior, you must modify the Lambda code.

Get the CloudFormation template

To use custom resources with an Amazon S3 bucket, save the following JSON or YAML template as a file.

JSON:

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Description": "Working with custom resources and S3",
  "Parameters": {
    "S3BucketName": {
      "Type": "String",
      "Description": "S3 bucket to create.",
      "AllowedPattern": "[a-zA-Z][a-zA-Z0-9_-]*"
    },
    "DirsToCreate": {
      "Description": "Comma delimited list of directories to create.",
      "Type": "CommaDelimitedList"
    }
  },
  "Resources": {
    "SampleS3Bucket": {
      "Type": "AWS::S3::Bucket",
      "Properties": {
        "BucketName": {"Ref":"S3BucketName"}
      }
    },
    "S3CustomResource": {
      "Type": "Custom::S3CustomResource",
      "DependsOn":"AWSLambdaExecutionRole",
      "Properties": {
        "ServiceToken": {"Fn::GetAtt": ["AWSLambdaFunction","Arn"]},
        "the_bucket": {"Ref":"SampleS3Bucket"},
        "dirs_to_create": {"Ref":"DirsToCreate"}
      }
    },
    "AWSLambdaFunction": {
      "Type": "AWS::Lambda::Function",
      "Properties": {
        "Description": "Work with S3 Buckets!",
        "FunctionName": {"Fn::Sub":"${AWS::StackName}-${AWS::Region}-lambda"},
        "Handler": "index.handler",
        "Role": {"Fn::GetAtt": ["AWSLambdaExecutionRole","Arn"]},
        "Timeout": 360,
        "Runtime": "python3.9",
        "Code": {
          "ZipFile": "import boto3\r\nimport cfnresponse\r\ndef handler(event, context):\r\n    # Init ...\r\n    the_event = event['RequestType']\r\n    print(\"The event is: \", str(the_event))\r\n    response_data = {}\r\n    s_3 = boto3.client('s3')\r\n    # Retrieve parameters\r\n    the_bucket = event['ResourceProperties']['the_bucket']\r\n    dirs_to_create = event['ResourceProperties']['dirs_to_create']\r\n    try:\r\n        if the_event in ('Create', 'Update'):\r\n            print(\"Requested folders: \", str(dirs_to_create))\r\n            for dir_name in dirs_to_create:\r\n                print(\"Creating: \", str(dir_name))\r\n                s_3.put_object(Bucket=the_bucket,\r\n                                Key=(dir_name\r\n                                    + '\/'))\r\n        elif the_event == 'Delete':\r\n            print(\"Deleting S3 content...\")\r\n            b_operator = boto3.resource('s3')\r\n            b_operator.Bucket(str(the_bucket)).objects.all().delete()\r\n        # Everything OK... send the signal back\r\n        print(\"Operation successful!\")\r\n        cfnresponse.send(event,\r\n                        context,\r\n                        cfnresponse.SUCCESS,\r\n                        response_data)\r\n    except Exception as e:\r\n        print(\"Operation failed...\")\r\n        print(str(e))\r\n        response_data['Data'] = str(e)\r\n        cfnresponse.send(event,\r\n                        context,\r\n                        cfnresponse.FAILED,\r\n                        response_data)"
        }
      }
    },
    "AWSLambdaExecutionRole": {
      "Type": "AWS::IAM::Role",
      "Properties": {
        "AssumeRolePolicyDocument": {
          "Statement": [
            {
              "Action": [
                "sts:AssumeRole"
              ],
              "Effect": "Allow",
              "Principal": {
                "Service": [
                  "lambda.amazonaws.com"
                ]
              }
            }
          ],
          "Version": "2012-10-17"
        },
        "Path": "/",
        "Policies": [
          {
            "PolicyDocument": {
              "Statement": [
                {
                  "Action": [
                    "logs:CreateLogGroup",
                    "logs:CreateLogStream",
                    "logs:PutLogEvents"
                  ],
                  "Effect": "Allow",
                  "Resource": "arn:aws:logs:*:*:*"
                }
              ],
              "Version": "2012-10-17"
            },
            "PolicyName": {"Fn::Sub": "${AWS::StackName}-${AWS::Region}-AWSLambda-CW"}
          },
          {
            "PolicyDocument": {
              "Statement": [
                {
                  "Action": [
                    "s3:PutObject",
                    "s3:DeleteObject",
                    "s3:List*"
                  ],
                  "Effect": "Allow",
                  "Resource": [
                    {"Fn::Sub": "arn:aws:s3:::${SampleS3Bucket}/*"},
                    {"Fn::Sub": "arn:aws:s3:::${SampleS3Bucket}"}
                  ]
                }
              ],
              "Version": "2012-10-17"
            },
            "PolicyName": {"Fn::Sub":"${AWS::StackName}-${AWS::Region}-AWSLambda-S3"}
          }
        ],
        "RoleName": {"Fn::Sub":"${AWS::StackName}-${AWS::Region}-AWSLambdaExecutionRole"}
      }
    }
  }
}

YAML:

Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.

Permission is hereby granted, free of charge, to any person obtaining a copy of this
software and associated documentation files (the "Software"), to deal in the Software
without restriction, including without limitation the rights to use, copy, modify,
merge, publish, distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

AWSTemplateFormatVersion: 2010-09-09
Description: Working with custom resources and S3
Parameters:
  S3BucketName:
    Type: String
    Description: "S3 bucket to create."
    AllowedPattern: "[a-zA-Z][a-zA-Z0-9_-]*"
  DirsToCreate:
    Description: "Comma delimited list of directories to create."
    Type: CommaDelimitedList
Resources:
  SampleS3Bucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Ref S3BucketName
  S3CustomResource:
    Type: Custom::S3CustomResource
    Properties:
      ServiceToken: !GetAtt AWSLambdaFunction.Arn
      the_bucket: !Ref SampleS3Bucket
      dirs_to_create: !Ref DirsToCreate
  AWSLambdaFunction:
     Type: "AWS::Lambda::Function"
     Properties:
       Description: "Work with S3 Buckets!"
       FunctionName: !Sub '${AWS::StackName}-${AWS::Region}-lambda'
       Handler: index.handler
       Role: !GetAtt AWSLambdaExecutionRole.Arn
       Timeout: 360
       Runtime: python3.9
       Code:
         ZipFile: |
          import boto3
          import cfnresponse
          def handler(event, context):
              # Init ...
              the_event = event['RequestType']
              print("The event is: ", str(the_event))
              response_data = {}
              s_3 = boto3.client('s3')
              # Retrieve parameters
              the_bucket = event['ResourceProperties']['the_bucket']
              dirs_to_create = event['ResourceProperties']['dirs_to_create']
              try:
                  if the_event in ('Create', 'Update'):
                      print("Requested folders: ", str(dirs_to_create))
                      for dir_name in dirs_to_create:
                          print("Creating: ", str(dir_name))
                          s_3.put_object(Bucket=the_bucket,
                                         Key=(dir_name
                                              + '/'))
                  elif the_event == 'Delete':
                      print("Deleting S3 content...")
                      b_operator = boto3.resource('s3')
                      b_operator.Bucket(str(the_bucket)).objects.all().delete()
                  # Everything OK... send the signal back
                  print("Operation successful!")
                  cfnresponse.send(event,
                                   context,
                                   cfnresponse.SUCCESS,
                                   response_data)
              except Exception as e:
                  print("Operation failed...")
                  print(str(e))
                  response_data['Data'] = str(e)
                  cfnresponse.send(event,
                                   context,
                                   cfnresponse.FAILED,
                                   response_data)
  AWSLambdaExecutionRole:
     Type: AWS::IAM::Role
     Properties:
       AssumeRolePolicyDocument:
         Statement:
         - Action:
           - sts:AssumeRole
           Effect: Allow
           Principal:
             Service:
             - lambda.amazonaws.com
         Version: '2012-10-17'
       Path: "/"
       Policies:
       - PolicyDocument:
           Statement:
           - Action:
             - logs:CreateLogGroup
             - logs:CreateLogStream
             - logs:PutLogEvents
             Effect: Allow
             Resource: arn:aws:logs:*:*:*
           Version: '2012-10-17'
         PolicyName: !Sub ${AWS::StackName}-${AWS::Region}-AWSLambda-CW
       - PolicyDocument:
           Statement:
           - Action:
             - s3:PutObject
             - s3:DeleteObject
             - s3:List*
             Effect: Allow
             Resource:
             - !Sub arn:aws:s3:::${SampleS3Bucket}/*
             - !Sub arn:aws:s3:::${SampleS3Bucket}
           Version: '2012-10-17'
         PolicyName: !Sub ${AWS::StackName}-${AWS::Region}-AWSLambda-S3
       RoleName: !Sub ${AWS::StackName}-${AWS::Region}-AWSLambdaExecutionRole

Deploy your CloudFormation template

To deploy your CloudFormation template, use the CloudFormation console or the AWS Command Line Interface (AWS CLI).

Note: If you receive errors when you run AWS CLI commands, then see Troubleshoot AWS CLI errors. Also, make sure that you're using the most recent AWS CLI version.

To use the CloudFormation console, complete the following steps:

1. Open the CloudFormation console.
2. Choose Create stack, and then choose With new resources (standard).
3. In Specify template, choose Upload a template file.
4. Choose Choose file, select the downloaded template, and then choose Next.
5. In Parameters, for S3BucketName, specify an S3 bucket name for the new bucket.
6. For DirsToCreate, enter a comma-delimited list of folders and sub-folders that you want to create.
   Note: For example, enter dir_1,dir_2/sub_dir_2,dir_3 as a list.
7. Complete the remaining steps in the setup wizard, and then choose Create stack.

To use the AWS CLI, complete the following steps:

1. Rename the downloaded template to custom-resource-lambda-s3.template.
2. Open a command line in your operating system (OS), and then navigate to the folder where you saved the template.
3. To deploy the template, run the create-stack command:

   aws cloudformation create-stack \                   
                      --timeout-in-minutes 10 \
                      --disable-rollback \
                      --stack-name testing-custom-resource-s3 \
                      --template-body file://custom-resource-lambda-s3.template \
                      --capabilities CAPABILITY_NAMED_IAM \
                      --parameters \
                      ParameterKey=DirsToCreate,ParameterValue="dir_1\,dir_2/sub_dir_2\,dir_3" \
                      ParameterKey=S3BucketName,ParameterValue="test-bucket-custom-resource"

   Note: Replace custom-resource-lambda-s3.template with your template, test-bucket-custom-resource with the S3 bucket name, and dir_1\,dir_2/sub_dir_2\,dir_3 with the list of folders and sub-folders that you want to create.

Related information

What are some best practices for implementing Lambda backed custom resources with CloudFormation?

How do I delete a Lambda backed custom resource that's stuck in DELETE_FAILED status or DELETE_IN_PROGRESS status in CloudFormation?