How do I troubleshoot a CloudWatch metric stream that doesn't forward metrics to the destination?

Short description

CloudWatch metric streams might not forward in the following situations:

• The metric stream status isn't running
• Metric stream filters exclude the metrics you want to forward
• The metrics are older than two days
• AWS Identity and Access Management (IAM) permissions are missing or incorrect
• The Amazon Data Firehose delivery stream doesn't have the correct configuration

Resolution

Note: If you receive errors when you run AWS Command Line Interface CLI (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Verify the metric stream status

AWS CloudWatch console

Complete the following steps:

1. Open the CloudWatch console.
2. In the navigation pane, choose Metrics, and then choose Streams.
3. Select your metric stream, and then verify that the Status shows Running.

If the status shows Stopped, then a user stopped the stream. To temporarily restart the stream, choose Actions, and then choose Start. For more information, see Metric stream operation and maintenance.

Note: When you stop and restart a metric stream, CloudWatch doesn't backfill metric data. CloudWatch only streams metrics published while the stream is running.

AWS CLI

Run the following stop metric-streams command:

aws cloudwatch stop-metric-streams --names STREAM_NAME

Note: Replace STREAM_NAME with the stream name.

Verify metric stream filters

If the metric stream status is Running, then verify that your metric stream filters don't exclude the metrics you want to forward.

Complete the following steps:

1. Run the following get-metric-stream command to check the filters in your metric stream configuration:

   aws cloudwatch get-metric-stream --name METRIC-STREAM-NAME

   Note: Replace METRIC-STREAM-NAME with the stream name.

2. In the output, check the IncludeFilters and ExcludeFilters sections.
   Note: The IncludeFilters filter lists the namespaces and metric names that the metric stream forwards. If this section is empty, then the metric stream forwards all metrics except those in ExcludeFilters. The ExcludeFilters filter lists the namespaces and metric names that the metric stream doesn't forward. If your metrics appear in this section, then they don't forward to your destination.

3. Verify that the metrics you want to forward aren't listed in ExcludeFilters.

Verify metric age

CloudWatch doesn't stream metrics published more than two days in the past. To verify that your metrics are recent enough to stream, display the metric as a graph in the CloudWatch console and check the timestamp of the most recent data point. If the most recent data point is more than two days old, then CloudWatch won't stream that metric.

Monitor metric stream metrics

CloudWatch publishes metrics about your metric stream to the AWS/CloudWatch/MetricStreams namespace. Monitor these metrics to identify delivery issues.

Complete the following steps:

1. Open the CloudWatch console.
2. In the navigation pane, choose Metrics, and then choose All metrics.
3. Select the AWS/CloudWatch/MetricStreams namespace.
4. Review the MetricUpdate, TotalMetricUpdate, and PublishErrorRate metrics.

MetricUpdate shows the number of metrics the stream successfully updated. TotalMetricUpdate shows the total count of metric updates the stream attempted. PublishErrorRate shows the rate of publishing errors. If the PublishErrorRate metric shows errors, then verify that the destination for the Firehose delivery stream exists and is correctly configured. If MetricUpdate is significantly lower than TotalMetricUpdate, then delivery issues occur.

Verify IAM permissions

The IAM role that you specify in the metric stream configuration must grant CloudWatch permissions to write to Firehose. Open the CloudWatch console, locate your metric stream, and then note the Amazon Resource Name (ARN) of the IAM role. Then open the IAM console and review the role. Verify that the role has a trust policy that allows the CloudWatch service principal streams.metrics.cloudwatch.amazonaws.com to assume the role. Also verify that the role has permissions to write to Firehose.

Verify Firehose delivery stream configuration

Take the following actions:

Open the Firehose console and review your delivery stream's monitoring metrics. Check whether the IncomingBytes and IncomingRecords metrics show data. If these metrics don't show data, then the delivery stream doesn't receive data from the metric stream.

Review the destination settings in the delivery stream configuration to confirm that they are correct. Then, turn on error logging to get detailed information about delivery failures. After you turn on error logging, check the CloudWatch Logs log group for error messages.

Related information

Troubleshooting metric streams in CloudWatch

Set up a metric stream

Metric stream operation and maintenance

Custom setup with Firehose