How do I troubleshoot a failed log delivery by a subscription filter in CloudWatch?

Short description

When you start streaming, check the CloudWatch Logs metrics to make sure that the filter pattern is correct and aligns with the incoming log events. The following are the most common metrics to review for troubleshooting:

• ForwardedBytes: The volume of log events in compressed bytes that are forwarded to the subscription destination.
• ForwardedLogEvents: The number of log events that are forwarded to the subscription destination.
• DeliveryErrors: The number of log events that CloudWatch Logs received an error for when data was forwarded to the subscription destination.
• DeliveryThrottling: The number of log events that indicate that CloudWatch Logs was throttled when data was forwarded to the subscription destination.

Note: If the destination service receives a retriable error, such as throttling or service errors, then CloudWatch Logs tries to send the data for 24 hours. If the error isn't retriable, such as AccessDenied or ResourceNotFound errors, then CloudWatch Logs doesn't try further delivery.

Resolution

Troubleshoot your failed log delivery by a subscription filter based on the destination service that you use.

Amazon Kinesis Data Streams

To troubleshoot a failed log delivery by a subscription filter to Amazon Kinesis Data Streams, complete the following tasks:

• Check that your Kinesis data stream is in an active state. To view the status, use either the Kinesis console or the DescribeStream API call.

• Check that the CloudWatch log group and Kinesis data stream are in the same AWS Region.

• Check the AWS Identity and Access Management (IAM) role that's linked to the subscription filter. Make sure that CloudWatch Logs has the necessary permissions to deposit data into your stream. The following is an example role permissions policy:
  Note: Replace example-region with your Region, example-account-id with your AWS account ID, and example-stream-name with the name of your stream.

  {  
    "Statement": [{  
      "Effect": "Allow",  
      "Action": "kinesis:PutRecord",  
      "Resource": "arn:aws:kinesis:example-region:example-account-id:stream/example-stream-name"  
    }]  
  }

• Check that the IAM role is configured with the appropriate trust policy. The following is an example trust policy:
  Note: Replace example-region with your Region and example-account-id with your account ID.

  {  
    "Statement": {  
      "Effect": "Allow",  
      "Principal": {  
        "Service": "logs.region.amazonaws.com"  
      },  
      "Action": "sts:AssumeRole",  
      "Condition": {  
        "StringLike": {  
          "aws:SourceArn": "arn:aws:logs:example-region:example-account-id:*"  
        }  
      }  
    }  
  }

• If you have a dedicated Kinesis data stream, then check the stream metrics to confirm that it functions. For more information, see Monitoring the Amazon Kinesis Data Streams Service with Amazon CloudWatch.

• If you experience issues with cross-account logging, then see How do I troubleshoot cross-account logging issues in CloudWatch?

Amazon Data Firehose

To troubleshoot a failed log delivery by a subscription filter to Amazon Data Firehose, complete the following tasks:

• Check that your Firehose stream is in an active state. To view the status, use either the Kinesis console or the DescribeDeliveryStream API call. If you use the data transformation feature, then make sure that the specified AWS Lambda function exists.

• Check the Firehose metrics to confirm that data is streaming into Firehose. Check metrics such as, IncomingBytes, IncomingRecords, DataReadFromKinesisStream.Bytes, and DataReadFromKinesisStream.Records.

• If Firehose isn't receiving data, then check the API-level CloudWatch metrics. The issue might originate from upstream. APIs, such as PutRecord and PutRecordBatch, send data to Firehose and must be correctly called.

• Check the error logs to review details about why the delivery failed. Make sure that the related IAM policy includes the logs:PutLogEvents permission. The following is an example IAM policy with the logs:PutLogEvents permission:
  Note: Replace example-region with your Region and example-account-id with your account ID.

  {  
      "Sid": "",  
      "Effect": "Allow",  
      "Action": [  
          "logs:PutLogEvents"  
      ],  
      "Resource": [  
          "arn:aws:logs:example-region:example-account-id:log-group:/aws/kinesisfirehose/Delivery_Stream:log-stream:*",  
          "arn:aws:logs:example-region:example-account-id:log-group:%FIREHOSE_POLICY_TEMPLATE_PLACEHOLDER%:log-stream:*"  
      ]  
  }

• Check that the IAM role that's linked to your Firehose stream has the appropriate permissions to allow CloudWatch Logs to deposit data. The following is an example permissions policy:
  Note: Replace example-region with your Region, example-account-id with your account ID, and example-stream-name with the name of your stream.

  {  
    "Statement": [{  
      "Effect": "Allow",  
      "Action": "firehose:PutRecord",  
      "Resource": "arn:aws:firehose:example-region:example-account-id:deliverystream/example-stream-name"  
    }]  
  }

• Check that the IAM role is configured with the appropriate trust policy. The following is an example trust policy:
  Note: Replace example-region with your Region and example-account-id with your account ID.

  {  
    "Statement": {  
      "Effect": "Allow",  
      "Principal": { "Service": "logs.amazonaws.com" },  
      "Action": "sts:AssumeRole",  
      "Condition": {   
           "StringLike": {   
               "aws:SourceArn": "arn:aws:example-region:example-account-id:*"  
           }   
       }  
    }  
  }

• If you experience issues with cross-account logging, then see How do I troubleshoot cross-account logging issues in CloudWatch?

• For issues that are related to different destinations, see Troubleshooting Amazon Data Firehose.

Lambda

To troubleshoot a failed log delivery by a subscription filter to Lambda, complete the following tasks:

• Check that the Lambda function has the required resource-based policy to grant CloudWatch Logs the permission to run your function. The following is an example policy statement:
  Note: Replace example-region, example-account-id, example-lambda-function, and example-log-name with your Region, account ID, Lambda function, and log name. 

  "Statement": [  
      {  
        "Sid": "",  
        "Effect": "Allow",  
        "Principal": {  
          "Service": "logs.amazonaws.com"  
        },  
        "Action": "lambda:InvokeFunction",  
        "Resource": "arn:aws:lambda:example-region:example-account-id:function:example-lambda-function",  
        "Condition": {  
          "StringEquals": {  
            "AWS:SourceAccount": "example-account-id"  
          },  
          "ArnLike": {  
            "AWS:SourceArn": "arn:aws:logs:example-region:example-account-id:log-group:example-log-name:*"  
          }  
        }  
      }  
    ] 

• Check that the Lambda function has an IAM role that has a trust relationship for lambda.amazonaws.com with the following role policy:

  "Statement": [  
      {  
          "Effect": "Allow",  
          "Principal": {  
          "Service": "lambda.amazonaws.com"  
          },  
          "Action": "sts:AssumeRole"  
      }  
  ]

OpenSearch Service

To troubleshoot a failed log delivery by a subscription filter to Amazon OpenSearch Service, complete the following tasks:

• Check that the Lambda function has an IAM role that has a trust relationship for lambda.amazonaws.com with the following role policy:
  Note: Replace example-region with your Region, example-account-id with your account ID, and example-target-domain with the name of the target domain.

  {"Version": "2012-10-17",  
    "Statement": [  
      {  
        "Action": [  
          "es:*"  
        ],  
        "Effect": "Allow",  
        "Resource": "arn:aws:es:example-region:exampleaccount-id:domain/example-target-domain/*"  
      }  
    ]  
  }

• If you experience issues with cross-account logging, then see How do I troubleshoot cross-account logging issues in CloudWatch?

• To troubleshoot CloudWatch Logs with OpenSearch Service subscription filter issues, see How do I troubleshoot CloudWatch Logs so that it streams to my OpenSearch Service domain?

Related information

How do I create, configure, and troubleshoot a subscription filter to Kinesis using the CloudWatch console?

How do I troubleshoot a data delivery failure between Amazon Data Firehose and Amazon S3?

How can I configure a CloudWatch subscription filter to invoke my Lambda function?