Can I use packetbeat or some other mechanism to monitor search queries hitting my AWS OpenSearch cluster?

0

Hi, I'm trying to find a good way to monitor search queries hitting our AWS OpenSearch cluster.

Use case: We have dozens of different applications that send queries to the cluster, some in response to user requests, some based on a cron schedule or other background activities. I need a way to analyze the volume of different types of queries being processed, average elapsed time for each type of query (e.g. by index and/or some normalized query pattern).

The most promising option I've found searching on the web for this problem is packetbeat. See https://www.elastic.co/blog/monitoring-the-search-queries for details on this setup. So, I'm wondering:

  • Is it possible to monitor AWS OpenSearch cluster using packetbeat?
  • Is there an alternate or better way to achieve what I described above?

I'm aware of slow query logging. This is somewhat helpful but not sufficient for my needs.

We are currently running elasticsearch version 6.8, so ideally looking for a solution that is compatible with that version, but we are also planning to upgrade next year to a newer version so I am interested in solutions that work with newer versions as well. Thanks!

satwood
질문됨 7달 전265회 조회
1개 답변
1

Hello,

On checking the link which you have shared as per my understanding the solution cannot be implemented in case of AWS OpenSearch service. This is because in start sniffing step I can see the instructions of installing Packetbeat to the nodes of production cluster which is unfortunately not possible due to managed nature of OpenSearch service.

Unfortunately, other than slow logs currently I do not see any other AWS recommended solution for monitoring slow queries. There are metrics like searchLatency in cloudwatch metrics which in combination of slow logs may help.

[+] Monitoring OpenSearch cluster metrics with Amazon CloudWatch - https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-cloudwatchmetrics.html

AWS
지원 엔지니어
Aman_A
답변함 7달 전
  • Thank you. This was my suspicion as well. I might look into feasibility of adding some instrumentation on the client side and/or sending requests through a proxy server where I could install packetbeat.

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인