How do I troubleshoot alarms that I configured on metric filters in CloudWatch Logs?

Short description

If you incorrectly configured CloudWatch alarms on metric filters, then you might experience the following issues:

• The alarm triggers, but you didn't receive an Amazon Simple Notification Service (Amazon SNS) notification.
• The alarm doesn’t trigger.
• Alarm shows Insufficient Data because of missing metrics.

To resolve these issues, check the CloudWatch alarm history to determine the cause. Then, apply one of the following resolutions appropriate for your scenario.

Resolution

Check alarm history

To determine whether your alarm triggers, complete the following steps:

1. Open the CloudWatch console.
2. In the navigation pane, choose Alarms.
3. Select your alarm.
4. Choose the History tab, and then review the alarm state changes and trigger history.

Then, troubleshoot your alarm in the following sections based on the alarm history.

Alarm triggered but you didn't receive an SNS notification

If the alarm history shows that the alarm triggered and changed to ALARM state, but you didn't receive notifications, check your alarm action configuration. Verify that the alarm action is Enabled, and then confirm that you configured the correct SNS topic Amazon Resource Name (ARN).

Then, review the following sections to troubleshoot the SNS configurations.

Troubleshoot SNS access policy restrictions

If the SNS access policy restricts access from CloudWatch to the SNS topic, then you receive the following error message:

"Failed to execute action arn:aws:sns:us-east-1:ACCOUNT_ID:TOPIC_NAME. Received error: CloudWatch Alarms is not authorized to perform: SNS:Publish on resource: arn:aws:sns:us-east-1:ACCOUNT_ID:TOPIC_NAME"

To resolve this error, complete the following steps:

1. Open the Amazon SNS console.
2. In the navigation pane, choose Topics.
3. Select your SNS topic.
4. Choose Edit.
5. Expand the Access policy section.
6. To grant CloudWatch permission to publish messages, add the following permissions statement to your access policy:

   { 
   "Sid": "Allow_Publish_Alarms", 
   "Effect": "Allow", 
   "Principal": { 
   "Service": "cloudwatch.amazonaws.com" 
   }, 
   "Action": "sns:Publish", 
   "Resource": "arn:aws:sns:example-region:example-account-id:example-topic" 
   }

   Note: Replace example-region with your AWS Region, example-account-id with your account ID, and example-topic with your topic name.
7. Choose Save changes.

Troubleshoot SNS topic encryption issues

If your SNS topic uses the default AWS Key Management Service (AWS KMS) alias/aws/sns key for server-side encryption, then you might receive the following error:

"CloudWatch Alarms does not have authorization to access the SNS topic encryption key"

Note: The default AWS KMS key policy for Amazon SNS doesn't allow CloudWatch to perform kms:Decrypt and kms:GenerateDataKey API operations.

To resolve this error, complete the following steps:

1. Complete the steps in the SNS topic encryption issues section of Why didn't I receive an Amazon SNS notification for my CloudWatch alarm trigger?

2. Open the Amazon SNS Console.

3. In the navigation pane, choose Topics.

4. Select your topic.

5. Choose Edit in the top right corner.

6. Expand the Encryption section.

7. Select your new customer managed key from the dropdown list.

8. Choose Save changes.

Alarm didn't trigger

If the alarm never enters ALARM state, then review the alarm configuration settings.

Complete the following steps:

1. Open the CloudWatch console.
2. In the navigation pane, choose Alarms.
3. Select your alarm.
4. Choose the Details tab to review the configuration.
5. Review the following configuration settings:
   Verify that the GreaterThanThreshold and LessThanThreshold match your intended values.
   Check that the value of Datapoints to Alarm (M) matches or is greater than the value of the Evaluation Period (N).
   Confirm that the alarm monitors the correct metric name, namespace, and dimensions.
   Note: If you incorrectly provide a value for one of these parameters, then CloudWatch tries to retrieve a metric that doesn't exist, and you get an empty dataset.
   Verify that the Treat missing data as setting aligns with your monitoring needs.
   Note: For immediate detection, use M = N. To reduce false positives, use M < N.

Alarm didn't trigger because of missing metrics

If the alarm shows Insufficient Data or the custom metric doesn't have data points, then troubleshoot metric filter configuration.

Verify metric filter exists and check configuration

Complete the following steps:

1. Open the CloudWatch console.
2. In the navigation pane, choose Log Management
3. Select your log group
4. Choose the Metric filters tab.
5. Verify that the metric filter exists.

Check the IncomingLogEvents metric and data points

To check IncomingLogEvents metric, complete the following steps:

1. Open the CloudWatch console
2. In the navigation pane, choose All metrics.
3. Select the AWS/Logs namespace.
4. Find IncomingLogEvents for your log group.
5. Set time range from filter creation to present.
6. Check if log events have been received.

To check your custom metric, complete the following steps:

1. In the navigation pane, choose All metrics.
2. Select your custom namespace.
3. Find your custom metric name.
4. Set the time range from filter creation to present.
5. Check that your data points exist.

If IncomingLogEvents is 0 and the custom metric doesn't have data, then logs don't generate. Verify that you turned on application logging and that you correctly configured the CloudWatch agent and SDK. Make sure that the AWS Identity and Access Management (IAM) permissions include logs:CreateLogStream and logs:PutLogEvents. Then, verify that the network connectivity allows HTTPS traffic to CloudWatch endpoints. For more information, see Troubleshooting the CloudWatch agent.

If IncomingLogEvents is less than 0 but custom metric doesn't have data, then you have a pattern mismatch. Review the log structure in your log group and examine actual log events in CloudWatch console. Copy the sample events to test patterns and then, verify the metric filter configuration.

If both IncomingLogEvents and your custom metric show data, then verify the metric filter configuration. Complete the following steps:

1. Open the CloudWatch console.
2. In the navigation pane, choose Log Management.
3. Choose your log group.
4. In the Metric filters tab, select your metric filter.
5. Choose Test pattern.
6. Select a log stream to test against. Or, paste actual log events from your log group.
7. Verify that the pattern matches the expected log events.

If the pattern doesn't match, then take the following actions:

• Verify that you correctly enter the field names and values. Field names are case-sensitive.
• Confirm that you used the $ symbol before field names.
• Make sure that you use quotes around string values.
• Use correct field positioning and spacing.
• Use && for AND conditions and || for OR conditions.

Configure metric filter values and default values

To make sure that CloudWatch reports data even when matches don't occur, configure a default value to prevents gaps in metrics.

Complete the following steps:

1. Open the CloudWatch console
2. In the navigation pane, choose Log Management.
3. Choose your log group.
4. In the Metric filters tab, select your metric filter.
5. Choose Edit and then, choose Next.
6. Update Metric Value and Default Value.
7. Choose Next and then, choose Save changes.

Related information

How do I troubleshoot a CloudWatch alarm that doesn't invoke?

Why didn't I receive an Amazon SNS notification for my CloudWatch alarm trigger?

Using Amazon CloudWatch alarms

Filter pattern syntax for metric filters