How do I use CloudWatch Logs to create a subscription filter to Kinesis Data Streams?

Resolution

Configure a subscription filter for a Kinesis data stream in the same account

If you don't already have one, then create a Kinesis data stream.

Note: The AWS Region for the CloudWatch log group and the Kinesis data stream destination must be the same.

Create an IAM policy and role with a custom trust policy

The AWS Identity and Access Management (IAM) role must have trust permissions for logs.yourregion.amazonaws.com and allow the permission kinesis:PutRecord.

Use a custom trust policy to create an IAM role. In the Custom trust policy section, enter the following trust policy:

{  "Statement": {
    "Effect": "Allow",
    "Principal": {
      "Service": "logs.region.amazonaws.com"
    },
    "Action": "sts:AssumeRole",
    "Condition": {
      "StringLike": {
        "aws:SourceArn": "arn:aws:logs:REGION:ACCOUNT_ID:*"
      }
    }
  }
}

Note: Replace REGION with your Region and ACCOUNT_ID with your AWS account ID.

In the Step 2: Add permissions section, choose Edit and then add the following policy statement to the custom policy:

{  "Statement": {    "Effect": "Allow",
    "Action": "kinesis:PutRecord",
    "Resource": "arn:aws:kinesis:REGION:ACCOUNT_ID:stream/STREAM_NAME"
  }
}

Note: Replace REGION with your Region, ACCOUNT_ID with your AWS account ID, and STREAM_NAME with your stream name.

The preceding policy is specific to Kinesis Data Streams. For more information about permissions for Amazon Data Firehose, see Grant access to your Firehose resources.

Create a subscription filter

Complete the following steps:

1. Open the CloudWatch console.
2. In the navigation pane, choose Log Management.
3. Select your Log group.
4. Choose Actions, and then Subscription Filters.
5. To configure the destination, choose Create Kinesis subscription filter.
6. Choose Current account.
7. Select your Kinesis data stream from the dropdown list.
8. Select the IAM role that you created.
9. Choose a distribution method. For more information about distributions, see Concepts.
10. In the Subscription filter pattern section, define the filter pattern.
11. Enter a name for your subscription filter.
12. Verify your pattern with the existing log event data.
13. Choose Start Streaming.

To verify that your data stream works, review the following subscription filter metrics:

• ForwardedBytes: The volume of log events in compressed bytes that are forwarded to the subscription destination.
• ForwardedLogEvents: The number of log events that are forwarded to the subscription destination.

To check that there aren't errors when you stream log events to your destination, review the following metrics:

• DeliveryErrors: The number of log events that show that CloudWatch Logs received an error for data that's forward to the subscription destination.
• DeliveryThrottling: The number of log events that show that CloudWatch Logs was throttled when data was forwarded to the subscription destination.

Configure a subscription filter for a Kinesis data stream in another account

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Create a destination 

In the data recipient's account, create a destination.

Create a subscription filter

In the source account, complete the following steps:

1. If you use an organization in AWS Organizations, then create an IAM role. Otherwise, proceed to step 2.
2. Add or validate the IAM permissions for the cross-account destination. 
3. Create a log group to send events to CloudWatch Logs.
4. Create the subscription filter.

To verify that your data stream works, review the following subscription filter metrics:

• ForwardedBytes: The volume of log events in compressed bytes that are forwarded to the subscription destination.
• ForwardedLogEvents: The number of log events that are forwarded to the subscription destination.

To check that there aren't errors when you stream log events to your destination, review the following metrics:

• DeliveryErrors: The number of log events that show that CloudWatch Logs received an error for data that's forward to the subscription destination.
• DeliveryThrottling: The number of log events that show that CloudWatch Logs was throttled when data was forwarded to the subscription destination.

If you have a dedicated Kinesis data stream, then check the data stream's metrics to confirm functionality.

For issues with cross-account logging, see Troubleshooting your CloudWatch cross-account setup.

Related information

Create an IAM policy and user

Cross-account cross-AWS Region log data sharing using Amazon Kinesis Data Streams