Create notification from logs

0

I would like to create a notification when a json file, that gets injected in my logs, gets updated.

We have a service that queries an external service and gets a response in json format. It queries this service every 5 minutes. Most of the time, the json is the same as the previous one, it only changes when it has an update. It is these updates that I would like to catch and trigger a notification.

I only have access to the logs where the json shows.

Would you have any suggestion on how to check that log and if the json has changed, then trigger a notification?

Miguel
已提問 24 天前檢視次數 167 次
1 個回答
1

Even with. this limited amount of information, there may be more than one way to skin this particular cat.

One approach (1) is to feed the cloudwatch logs to a Lambda function, Kinesis Data Streams or Firehose through a log group-level subscription filter, let it go through every log message to find the aforementioned JSON response, canonicalize and compare against a last-saved version and determine if it has changed to trigger a notification. On first sight this feels rather expensive in terms of effort and resource utilization.

Assuming that the external service is HTTP or network-based, another approach (2) would be to insert a proxy between the external service and your consumer, and perform the change-detection logic locally. And here we're talking about both an HTTP proxy and the actual proxy microservice design pattern, mind you. This component doesn't need to perform any transformations in the content, just to parrot a request, wait for the response and forward it blindly, and then canonicalize it, compare and trigger the notification when applicable. We're talking about a few lines of python, nodejs or perl in a Lambda. Depending on the level of control you have on the consumer, you may just (a) change the address for that single endpoint, (b) define an HTTP_PROXY environment variable or (c) install an iptables transparent proxy (like squid and mitmproxy do). Please note that (2.a) and (2.b) are a one-afternoon project, but (2.c) might be less cost-effective than approach 1, and it's fair to say that in any case you are adding a moving part that depending on the circumstances can become another possible point of failure.

profile picture
已回答 3 天前
  • Thanks Javier for the few approaches which have triggered some other ideas.

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南