How do I use AWS CloudFormation to automatically deploy a DataSync agent on Amazon EC2 with an AWS PrivateLink endpoint?

Resolution

Use AWS CloudFormation to automatically create and activate a DataSync agent that runs on an Amazon EC2 instance through an AWS PrivateLink endpoint.

Use the YAML template that uses a custom resource that uses a Lambda function to retrieve the activation key for your DataSync Agent. Copy the following YAML code, and then save it as your AWS CloudFormation template:

AWSTemplateFormatVersion: '2010-09-09'  
Description: >-  
  This CloudFormation template deploys an EC2 instance with the AWS DataSync  
  agent AMI.  
  
Metadata:  
  AWS::CloudFormation::Interface:  
    ParameterGroups:  
    - Label:  
        default: Deploy DataSync agent on Amazon EC2 via PrivateLink VPC endpoint  
      Parameters:  
      - InstanceType  
      - ImageId  
      - SubnetId  
      - VpcId  
    ParameterLabels:  
      InstanceType:  
        default: Agent EC2 Instance Size  
      SubnetId:  
        default: Select your Subnet  
      VpcId:  
        default: Select your VPC Contains Above Subnet  
      ImageId:  
        default: Agent Image (Keep default - Do NOT change)  
  
Parameters:  
  InstanceType:  
    Description: Instance type for the DataSync agent EC2 instance  
    Type: String  
    Default: m5.2xlarge  
    AllowedValues:  
      - m5.2xlarge  
      - m5.4xlarge  
    ConstraintDescription: must be a valid EC2 instance type.  
  
  ImageId:  
    Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>  
    Default: /aws/service/datasync/ami  
  
  SubnetId:  
    Type: AWS::EC2::Subnet::Id  
    Description: The subnet where to deploy the instance  
  
  VpcId:   
    Type: AWS::EC2::VPC::Id  
    Description: Select the VPC containing the subnet selected above.  
  
Resources:  
  NewKeyPair:  
    Type: 'AWS::EC2::KeyPair'  
    Properties:  
      KeyName: !Sub "ssh-key-cfn-stack-${AWS::StackName}"  
  
  DataSyncAgentInstance:  
    Type: AWS::EC2::Instance  
    Properties:  
      ImageId: !Ref ImageId  
      InstanceType: !Ref InstanceType  
      SubnetId: !Ref SubnetId  
      SecurityGroupIds:  
        - !Ref DataSyncAgentSecurityGroup  
      KeyName: !Ref NewKeyPair  
      Tags:  
        - Key: "Name"  
          Value: !Ref 'AWS::StackName'  
  
  DataSyncAgentSecurityGroup:  
    Type: AWS::EC2::SecurityGroup  
    Properties:  
      GroupDescription: Enable SSH access and DataSync agent communication  
      SecurityGroupIngress:  
        - IpProtocol: tcp  
          FromPort: 80  
          ToPort: 80  
          CidrIp: 0.0.0.0/0  
      VpcId: !Ref VpcId  
  
  DsInterfaceEndpoint:  
    Type: 'AWS::EC2::VPCEndpoint'  
    Properties:  
      VpcEndpointType: 'Interface'  
      ServiceName: !Sub 'com.amazonaws.${AWS::Region}.datasync'  
      VpcId: !Ref VpcId  
      SubnetIds:   
        - !Ref SubnetId  
      SecurityGroupIds:  
        - !Ref DsVpceSecurityGroup  
  
  DsVpceSecurityGroup:  
    Type: 'AWS::EC2::SecurityGroup'  
    Properties:  
      GroupDescription: 'Allow control traffic from agent'  
      VpcId: !Ref VpcId  
      SecurityGroupIngress:  
        - IpProtocol: tcp  
          FromPort: 443  
          ToPort: 443  
          CidrIp: !Sub "${DataSyncAgentInstance.PrivateIp}/32"  
        - IpProtocol: tcp  
          FromPort: 1024  
          ToPort: 1064  
          CidrIp: !Sub "${DataSyncAgentInstance.PrivateIp}/32"  
        - IpProtocol: tcp  
          FromPort: 22  
          ToPort: 22  
          CidrIp: !Sub "${DataSyncAgentInstance.PrivateIp}/32"  
  
  LambdaActivator:  
      Type: AWS::Lambda::Function  
      DependsOn: DsInterfaceEndpoint  
      Properties:  
        FunctionName: !Ref 'AWS::StackName'  
        Description: 'Lambda function to get the activation key'  
        VpcConfig:   
          SecurityGroupIds:  
            - !Ref DataSyncAgentSecurityGroup  
          SubnetIds:  
            - !Ref SubnetId  
        Environment:  
          Variables:  
            DsDnsEntryList: !Select [1, !Split [':', !Select [0, !GetAtt DsInterfaceEndpoint.DnsEntries]]]  
        Code:  
          ZipFile: !Sub |  
            import boto3, json, urllib.request, socket, time, os  
            import cfnresponse  
  
            print('Loading function')  
            def handler(event, context):  
                print('EVENT:')  
                print(event)  
                responseData = {}  
                try:  
                    if event['RequestType'] == "Create":  
                        agentIp = "${DataSyncAgentInstance.PrivateIp}"  
                        awsRegion = "${AWS::Region}"  
                        VpceDnsName = os.environ['DsDnsEntryList']  
                        VpceIp = socket.gethostbyname(VpceDnsName)  
  
                        print("Agent IP: " + agentIp)  
                        print("AWS Region: " + awsRegion)  
                        print("VPC Endpoint DNS Name: " + VpceDnsName)  
                        print("VPC IP: " + VpceIp)  
  
                        activateUrl = "http://" + agentIp + "/?gatewayType=SYNC&activationRegion=" + awsRegion + "&privateLinkEndpoint=" + VpceIp + "&endpointType=PRIVATE_LINK&no_redirect"  
                        print(activateUrl)  
  
                        time.sleep(30)  
                        print("Sending URL for Getting activation key")  
                        res = urllib.request.urlopen(urllib.request.Request(  
                            url=activateUrl,  
                            method='GET'),  
                            timeout=600)  
                          
                        print("HTTP Response :" + str(res.status) + " " + res.reason)  
                        activationKey = res.read().decode('utf-8')  
  
                        responseData['Data'] = activationKey  
                      
                    elif event['RequestType'] == "Delete":  
                        print("Lambda is being deleted by the CloudFormation stack!!")  
                      
                    else:  
                        print("Should not perform actions!!")  
  
                    cfnresponse.send(event, context, cfnresponse.SUCCESS,   
                            responseData, 'cfn-customresource-id')  
                      
                    return {  
                      'body': json.dumps('Execution completed!')  
                    }  
  
                except Exception as e:  
                    print(e)  
                    cfnresponse.send(event, context, cfnresponse.FAILED,   
                            responseData, 'cfn-customresource-id')  
                    raise e  
  
        Handler: index.handler  
        Role: !GetAtt FunctionIamRole.Arn  
        Runtime: python3.12  
        Timeout: 300  
  
  ActivatorInvoker:  
    DependsOn:  
      - LambdaActivator  
      - DataSyncAgentInstance  
    Type: AWS::CloudFormation::CustomResource  
    Properties:  
      ServiceToken: !GetAtt LambdaActivator.Arn  
  
  FunctionIamRole:  
    Type: AWS::IAM::Role  
    Properties:  
      AssumeRolePolicyDocument:  
        Version: '2012-10-17'  
        Statement:  
          - Effect: Allow  
            Principal:  
              Service:  
                - lambda.amazonaws.com  
            Action:  
              - 'sts:AssumeRole'  
      ManagedPolicyArns:  
        - "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"  
      Policies:   
        - PolicyName: !Sub "ec2-permissions-lambda-${AWS::StackName}"  
          PolicyDocument:  
            Version: '2012-10-17'  
            Statement:  
              - Effect: Allow  
                Action:  
                  - ec2:CreateNetworkInterface  
                  - ec2:DeleteNetworkInterface  
                  - ec2:DescribeNetworkInterfaces  
                Resource: "*"  
    
  ActiveAgent:  
    Type: AWS::DataSync::Agent  
    Properties:  
      ActivationKey: !GetAtt ActivatorInvoker.Data  
      AgentName: !Sub "datasync-agent-created-from-cfn-${AWS::StackName}"  
      VpcEndpointId: !GetAtt DsInterfaceEndpoint.Id  
      SecurityGroupArns:  
        - !Sub "arn:aws:ec2:${AWS::Region}:${AWS::AccountId}:security-group/${DsVpceSecurityGroup}"  
      SubnetArns:  
        - !Sub "arn:aws:ec2:${AWS::Region}:${AWS::AccountId}:subnet/${SubnetId}"  
  
Outputs:  
  KeyPairId:  
    Description: KeyPairId of the newly created SSH Key to connect the EC2 (you can download the key under Parameter Store, a capability of AWS Systems Manager)  
    Value: !GetAtt NewKeyPair.KeyPairId  
  
  PrivateIp:  
    Description: PrivateIp of the newly created EC2 instance  
    Value: !GetAtt DataSyncAgentInstance.PrivateIp  
  
  ActivationKey:  
    Description: Agent Activation Key  
    Value: !GetAtt ActivatorInvoker.Data

Use the AWS CloudFormation console to create an AWS CloudFormation stack. On the Create stack page, for Prerequisite - Prepare template, choose Choose an existing template. Under Specify template, choose Upload a template file, and then select the template that you saved.

Note: In the Parameters section, the private subnet that you specify must have access to an Amazon Simple Storage Service (Amazon S3) service through an Amazon Virtual Private Cloud (VPC) gateway endpoint.