Is it possible to have an alarm action get re-triggered every so often?

Question

I have an alarm on my SES Bounce Reputation that sends an SNS message when the threshold is breached.  However, this action only happens once and if I happen to miss that notification then I'm in danger of having my email processing impacted.

In the past we've used other notification systems that allow you to configure repeating alerts.  If you don't respond to the alert within some time period (ie. 6 hours) it triggers another notification.  Is this type of thing possible with CloudWatch Alarms?  How do you handle missed alert notifications?

On a related note; is it possible to trigger a CloudWatch alarm notification at a specific time?  For example, at 9am every morning, alert me if my SES bounce reputation is > 5%.  This would also solve the problem of missing a notification.  At least you would get another one the next day.

Jsc · Answer

Hello,
CloudWatch currently cannot be configured to periodically re-execute action(s) while an alarm remains in a given (e.g. ALARM) state.
There are a few ways to work around this limitation.

1. Your suggestion works

2. You could use a lambda to implement the action, configure the lambda to trigger on an alarm state change event, and from the lambda create a scheduled event that will re-trigger the lambda after a given interval if the alarm is still in ALARM state. Up to now, this is the most common way I've heard of to implement alarming recurrence.

3. I believe using metric math is now a simpler way in some cases (I think it works well for your 2nd case, at 9am every morning). You can create a metric math alarm that will "zero" your metric's value (or offset it below threshold) at given times/days, so your alarm will be forced into OK status over those times, and evaluate the real metric value only at given times/intervals. For example IF(HOUR(m1)>7 AND HOUR(m1)<9 AND MINUTE(m1)<45, m1, 0) will zero your value except every day between 8:00am and 8:45am - that's just an example, you might be interested by the following blog post https://aws.amazon.com/blogs/mt/enhance-cloudwatch-metrics-with-metric-math-functions/

The drawback is that it needs you to potentially create 2 alarms if you need to also monitor your metric continuously (as most of the time, this metric math alarm would not evaluate anything and show up as OK).

KangarooBox · Answer

I supposed you could re-trigger stale alarms.  I'm envisioning a Lambda that runs every morning looking for alarms that have been active more than X (6?) hours.  It could then update those alarms (SetAlarmState?) to force a re-trigger of the action.

It's not a very elegant solution and it seems like someone, somewhere would have run into this problem before.  Is there not a native solution to stale CW alarms?

Is it possible to have an alarm action get re-triggered every so often?

Relevant content