resetting knn circuit breaker triggered

0

How can we reset the knn circuit breaker when it is triggered. According to knn documentation we need to update a cluster setting but are for some reason denied access to this setting: knn.memory.circuit_breaker.limit

What is the work around?

iamlmt
asked 12 days ago57 views
2 Answers
0
Accepted Answer

Update on the answer, the "defaults" settings are not able to be changed but the "persistent" changes are able to be changed, the correct thing to do to evict and reset the KNN cache is to perform the following request:

HTTP PUT /_cluster/settings {"persistent":{"knn.memory.circuit_breaker.limit": null } }

iamlmt
answered 11 days ago
0

Hi, As you mentioned yo need to reset the k-NN circuit breaker when it is triggered in Amazon OpenSearch Service, and the k-NN circuit breaker is controlled by the knn.memory.circuit_breaker.limit cluster setting, with this setting you can determine the maximum amount of memory that can be used by the k-NN feature.

A "denied access" error when trying to update this setting, could mean that you do not have the necessary permissions to modify the cluster settings. You will need to request the appropriate IAM permissions from your AWS administrator. The required permissions are: "opensearch:UpdateDomainConfig" "opensearch:DescribeDomain" "opensearch:DescribeDomainConfig"

After you update the setting, the k-NN circuit breaker should be reset, and your queries should be able to execute without triggering the circuit breaker.

profile pictureAWS
Arifc
answered 12 days ago
  • Thank you for your response, however I don't believe that to be the issue after trying it and confirming all opensearch permissions are set on the cluster which has IP Address level security configuration.

    The IAM permissions are added and the response from the cluster is the same response. According to this: https://forum.opensearch.org/t/how-to-deal-with-knn-circuit-breaker-triggered-stays-set-nodes-at-max-cache-capacity/4733

    This request should return the value to its default and trigger a "cache rebuild where all entries are evicted".

    Request Body: HTTP PUT /_cluster/settings {"default":{"knn.memory.circuit_breaker.limit": null } }

    Response Body {"Message": "Your request: '/_cluster/settings' payload is not allowed."}

    If this cannot be set by some feature of OpenSearch then is there another way to cause a "cache rebuild where all entries are evicted" safely?

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions