Cloudwatch alerts - find keywords and alert - but notify based on server (stream) name

Question

Morning all, moving from Nagios to Cloudwatch and have a few test cases, but I am sure once I know how I can do it all, so for this one, I have a server farm of 7 servers.  They write a JSON log file every minute, and the basic output I look for today is either a status_ok, status_warning or status_critical.    I have my dev server setup, logs going into cloudwatch group and into the dev stream.

I setup a filter to look for that status_ok, if it doesn't see it in 5 minutes, alert and that worked perfect.  The problem is I put up my next server, same log group, different stream [server-1] for example.  The alert fired and of course said dev as I didn't realize the mettric filter is on the group not the stream.

So, basically I want the team to know if server-2 has an issue, let them get the alert that server is the one that didn't have the check.  So, what is the best way to search, filter and alert based on the stream and not whole group?

Thanks!

Accepted Answer

If I'm understanding you correctly what you might want to do is add a dimension to your Metric Filter which captures the server information. The dimension should show in your notification. (You could create a separate metric filter for each, but if you have a consistent format of the log extracting the information as a dimension will be much less overhead, and will work if you add more servers).

An example of adding a dimension which is extracted from the log data - https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/ExtractBytesExample.html.

Cloudwatch alerts - find keywords and alert - but notify based on server (stream) name

Relevant content