How do I stream data from CloudWatch Logs to a VPC-based OpenSearch Service cluster in a different AWS account?

Resolution

To stream data from CloudWatch Logs to an OpenSearch Service cluster in another account, create a VPC peering connection between AWS Lambda and OpenSearch Service. The VPC peering session allows Lambda to send data from CloudWatch Logs to your OpenSearch Service domain.

Note: In this article, Account A is the account that streams CloudWatch Logs and Account B contains the OpenSearch Service VPC.

Set up a VPC peering connection between Account A and Account B

Complete the following steps:

1. In Account A and in Account B, open the Amazon VPC console.
   Note: Make sure that your VPCs don't have overlapping CIDR blocks.
2. Create a VPC peering session between the two custom VPCs: Lambda for Account A and OpenSearch Service for Account B.
3. Update the route table for both VPCs.
4. In Account A, select an existing security group for the Lambda function, or create a new one. Then, add an outbound rule to allow traffic to OpenSearch Service subnets.
5. In Account B, select the OpenSearch Service security group, and then add an inbound rule to allow traffic from the Lambda subnets.

For more information about VPC peering connections, see Work with VPC peering connections.

Set up the Lambda function's IAM role and permissions

Complete the following steps:

1. In Account A, open the AWS Identity and Access Management (IAM) console.
2. Choose Roles, and then choose Create Role.
3. Configure the following role settings:
   For Trusted entity type, select AWS Service.
   For Use case, select Lambda.
4. Choose Next.
5. For Add permissions, enter AWSLambdaVPCAccessExecutionRole.
6. Choose Next.
7. Choose Create Role.
8. Select your role.
9. Under Permissions, choose Add permissions, and then choose Inline policy.
10. Enter the following policy:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "Statement1",
                "Effect": "Allow",
                "Action": [
                    "es:ESHttpPut",
                    "es:ESHttpPost"
                ],
                "Resource": "arn:aws:es:aws-region:aws-account-B:domain/opensearch-domain-name/*"
            }
        ]
    }

    Note: Replace aws-region with your AWS Region, aws-account-B with the Amazon Resource Name (ARN) of Account B, and opensearch-domain-name with your OpenSearch Service domain name.
    The preceding policy allows the OpenSearch Service domain in Account B to run Put and Post requests.

Set up the OpenSearch Service domain access policy

Complete the following steps:

1. In Account B, open the OpenSearch Service console.
2. Choose Actions.
3. Choose Edit Security configuration, and then for Access policy, enter the following policy:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "AWS": "arn:aws:iam::aws-account-A:role/lambda-role-name"
         },
         "Action": [
           "es:ESHttpPut",
           "es:ESHttpPost"
         ],
         "Resource": "arn:aws:es:aws-region:aws-account-B:domain/opensearch-domain-name/*"
       }
     ]
   }

   Note: Replace aws-account-A with the ARN of Account A and aws-account-B with the ARN of Account B. Replace lambda-role-name with your Lambda role, aws-region with your Region, and opensearch-domain-name with your OpenSearch Service domain name.
   The preceding policy allows OpenSearch Service to make calls from the Lambda function's execution role.

Set up CloudWatch Logs in Account A

Complete the following steps:

1. In Account A, open the CloudWatch console.
2. In the navigation pane, choose Log groups.
3. Select your log group.
4. Choose Actions.
5. Choose Subscription filters, and then choose Create Amazon OpenSearch Service subscription filter.
6. For Select account, select Another account.
7. For Amazon OpenSearch Service ARN, enter the OpenSearch Service ARN of Account B. For example: arn:aws:es:aws-region-aws-account-B:domain/opensearch-domain-name.
8. For Amazon OpenSearch Service endpoint, enter Account B's domain endpoint. For example: vpc-domain-name-some_id.us-east-1.es.amazonaws.com.
   Note: Don't include https://.
9. For Lambda IAM Execution Role, select your IAM role.
10. In Configure log format and filters, select your log format and subscription filter pattern.
11. For Subscription filter, enter a filter name, and then choose Start Streaming.

For more information about streaming, see Streaming CloudWatch Logs data to Amazon OpenSearch Service.

Configure the VPC for Lambda in Account A

Complete the following steps:

1. In Account A, open the Lambda console.
2. Select the Lambda function that you created to stream the log. For example: LogsToOpenSeach_domain-name_aws-account-b
3. Choose Configuration.
4. Choose VPC.
5. Under VPC, choose Edit.
6. Select your VPC, subnets, and security groups.
   Note: This selection allows the Lambda function to run inside a VPC. Lambda uses VPC routing to send data to the OpenSearch Service domain.
7. Choose Save.

For more information about Amazon VPC configurations, see Giving Lambda function to access resources in an Amazon VPC.

Verify your setup

Complete the following steps:

1. Open the Lambda console.
2. Check the Error count and success rate metric to verify that CloudWatch Logs is delivering the logs to OpenSearch Service.
3. Check the Indexing rate metric in OpenSearch Service to confirm that the data sends.

CloudWatch Logs can now stream across your accounts in your Amazon VPC.

Related information

How do I troubleshoot CloudWatch Logs so that it streams to my OpenSearch Service domain?