Can I use packetbeat or some other mechanism to monitor search queries hitting my AWS OpenSearch cluster?

0

Hi, I'm trying to find a good way to monitor search queries hitting our AWS OpenSearch cluster.

Use case: We have dozens of different applications that send queries to the cluster, some in response to user requests, some based on a cron schedule or other background activities. I need a way to analyze the volume of different types of queries being processed, average elapsed time for each type of query (e.g. by index and/or some normalized query pattern).

The most promising option I've found searching on the web for this problem is packetbeat. See https://www.elastic.co/blog/monitoring-the-search-queries for details on this setup. So, I'm wondering:

  • Is it possible to monitor AWS OpenSearch cluster using packetbeat?
  • Is there an alternate or better way to achieve what I described above?

I'm aware of slow query logging. This is somewhat helpful but not sufficient for my needs.

We are currently running elasticsearch version 6.8, so ideally looking for a solution that is compatible with that version, but we are also planning to upgrade next year to a newer version so I am interested in solutions that work with newer versions as well. Thanks!

satwood
已提問 7 個月前檢視次數 265 次
1 個回答
1

Hello,

On checking the link which you have shared as per my understanding the solution cannot be implemented in case of AWS OpenSearch service. This is because in start sniffing step I can see the instructions of installing Packetbeat to the nodes of production cluster which is unfortunately not possible due to managed nature of OpenSearch service.

Unfortunately, other than slow logs currently I do not see any other AWS recommended solution for monitoring slow queries. There are metrics like searchLatency in cloudwatch metrics which in combination of slow logs may help.

[+] Monitoring OpenSearch cluster metrics with Amazon CloudWatch - https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-cloudwatchmetrics.html

AWS
支援工程師
Aman_A
已回答 7 個月前
  • Thank you. This was my suspicion as well. I might look into feasibility of adding some instrumentation on the client side and/or sending requests through a proxy server where I could install packetbeat.

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南