How do I troubleshoot CloudWatch log event ingestion and visibility issues across different services?

Short description

Recently ingested events and updated metadata may not appear immediately in API responses, because CloudWatch Logs operates on an eventual consistency model. However, the events and metadata appear in queries that you make later.

The following log events and log stream metadata don't immediately update when Amazon CloudWatch ingests them:

• The lastEventTimestamp field in the DescribeLogStreams response doesn't detect new log events.
• CloudWatch Logs API operations, such as GetLogEvents, FilterLogEvents, DescribeLogStreams, and StartQuery, don't return recently ingested events or updated metadata when you call them after you ingest events.
• Consecutive API calls within short intervals can return incomplete results. For example, a short interval of 5 minutes can return incomplete results.

For real-time log processing and immediate delivery, use CloudWatch Logs subscription filters. You can also use filter patterns to reduce the volume of data forwarded to downstream services. For more information, see Real-time processing of log data with subscriptions.

Resolution

You can create subscription filters for log groups, or individual AWS accounts.

Take one of the following actions to create a log group-level subscription filter, based on your destination resource:

• For Amazon Kinesis Data Stream, see Example 1: Subscription filters with Amazon Kinesis Data Streams.
• For AWS Lambda, see Example 2: Subscription filters with AWS Lambda.
• For Amazon Data Firehose, see Example 3: Subscription filters with Amazon Data Firehose.

Take one of the following actions to create an account-level subscription filter, based on your destination resource:

• For Kinesis, see Example 1: Subscription filters with Amazon Kinesis Data Streams.
• For Lambda, see Example 2: Subscription filters with AWS Lambda.
• For Firehose, see Example 3: Subscription filters with Amazon Data Firehose.
  Note: Subscription filters deliver a real-time feed of log events to other AWS services. CloudWatch Logs base64 encodes the log events that it delivers through subscription filters and compresses them in gzip format.

To receive their log events from the owner of a different account on your AWS resources, see Cross-account cross-Region subscriptions.

Related information

CloudWatch Logs API Reference

CloudWatch Logs examples using SDK for Java 2.x