By using AWS re:Post, you agree to the AWS re:Post Terms of Use

Cloudwatch Subscription Filter does not ingest logs in Kinesis Data Stream

0

I have developed a web application using API Gateway and Lambda function. To capture and process logs from this application, i'm using cloudwatch logs and I set up a subscription filter for a Kinesis Data Stream. Despite having logs in CloudWatch each time the application is used, I am unable to see these logs in my Kinesis Data Stream.

1 Answer
0

Based on the details provided, it sounds like you have the logging from your API Gateway/Lambda application configured correctly to send logs to CloudWatch Logs, but the subscription filter to forward those logs from CloudWatch to Kinesis is not working as expected. A few things you could check: [1]

  • Verify the IAM role used by the subscription filter has the necessary permissions to read logs from the CloudWatch log group and write to the Kinesis stream. [2]
  • Check that the subscription filter is configured to match the correct log group and log stream prefix for your Lambda function logs.
  • Ensure no filters are applied in the subscription that may be excluding the log events.
  • Check for any errors in CloudWatch Logs that may provide clues, such as insufficient IAM permissions or connection issues to Kinesis.
  • Confirm the Kinesis stream has sufficient shard capacity to receive the additional log volume from CloudWatch.
  • Try reducing the log level in Lambda temporarily to send fewer logs as a test.

I'd also suggest using the AWS CLI or SDK to put a test record in the CloudWatch log group and verify it reaches the Kinesis stream, to isolate the subscription filter as the potential issue. Let me know if any of those suggestions help or if you have additional logs/configuration details that could provide more context.

Sources:

[1] Troubleshoot execution issues in Lambda - AWS Lambda

[2] Streaming Data Solution for Amazon Kinesis

profile pictureAWS
answered 8 months ago
profile picture
EXPERT
reviewed 8 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions