How do I resolve search or write rejections in OpenSearch Service?

7 minute read

I want to troubleshoot rejected search or write requests to my Amazon OpenSearch Service cluster.

Short description

When you write or search for data in your Amazon OpenSearch Service cluster, you might receive the following "HTTP 429" error or "es_rejected_execution_exception" message:

"error":"elastic: Error 429 (Too Many Requests): rejected execution of org.elasticsearch.transport.TransportService$7@b25fff4 on EsThreadPoolExecutor[bulk, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@768d4a66[Running, pool size = 2, active threads = 2, queued tasks = 200, completed tasks = 820898]] [type=es_rejected_execution_exception]"

This issue occurs for the following reasons:

You exceed your data node Amazon Elastic Compute Cloud (Amazon EC2) instance type's search or write quota.
Your cluster has high CPU utilization and JVM memory pressure.
You unevenly distributed shards in your OpenSearch Service cluster.
Your active and queue threads are full.

Resolution

Calculate the search and write quotas for your data node instance type

Your node queues requests when it exceeds its concurrent operation limit. Enter your vCPU count into the relevant formula to calculate this limit.

To calculate maximum active threads for search requests, use the following formula:

int ((available_processors * 3) / 2) + 1

To calculate maximum active threads for write requests, use the following formula:

int (available_processors)

Note: Replace available_processors with the number of vCPUs available in your node.

For example, you choose the R5.2xlarge node type for five nodes in your version 7.4 cluster. Each node has 8 vCPUs.

For each R5.2xlarge node, you can perform a maximum of 13 concurrent search operations:

(8 vCPUs * 3) / 2 + 1 = 13 concurrent search operations

Because the cluster has 5 nodes, you can perform a maximum of 65 search operations:

5 nodes * 13 concurrent search operations = 65 total search operations

For each R5.2xlarge node, you can perform a maximum of 8 concurrent write operations:

8 vCPUs = 8 concurrent write operations

Because the cluster has 5 nodes, you can perform a maximum of 40 write operations:

5 nodes * 8 concurrent write operations = 40 total write operations

To resolve this issue, reduce your concurrent or total search and write operations to stay within the quota that you calculated. For more information about vCPU counts for each node type, see OpenSearch Service pricing.

Check CloudWatch metrics for OpenSearch Service

To troubleshoot your 429 exception, check the following Amazon CloudWatch metrics for your cluster:

IndexingRate: Number of indexing operations per minute. Sudden spikes or sustained high values compared to capacity limits show pressure.
SearchRate: Total search requests per minute. High rates that approach node capacity limits show pressure.
CoordinatingWriteRejected: Rejections on the coordinating node because of indexing pressure. Any non-zero values show an issue. This metric is available in version 7.1 and later.
PrimaryWriteRejected: Rejections on primary shards because of indexing pressure. Any non-zero values show an issue. This metric is available in version 7.1 and later.
ReplicaWriteRejected: Rejections on replica shards because of indexing pressure. Any non-zero values show an issue. This metric is available in version 7.1 and later.
ThreadpoolWriteQueue: Queued tasks in the write thread pool. The maximum queue size is 10,000 for version 7.1 and later, or 200 for earlier versions.
ThreadpoolWriteRejected: Rejected tasks in the write thread pool. For version 7.1 and later, use this metric with CoordinatingWriteRejected, PrimaryWriteRejected, and ReplicaWriteRejected to monitor rejections.
ThreadpoolSearchQueue: Queued tasks in the search thread pool. The maximum search queue size is 1,000.
ThreadpoolSearchRejected: Rejected tasks in the search thread pool. If this number continually grows, then scale your cluster.

Also, check whether the JVMMemoryPressure CloudWatch cluster metric reaches 75%. If JVMMemoryPressure is consistently at 75% or higher, then you might encounter cluster performance issues. To troubleshoot this issue, see How do I troubleshoot high JVM memory pressure on my Amazon OpenSearch Service cluster?

Check whether the CPUUtilization CloudWatch metric reaches 80%. If CPUUtilization is consistently at 80% or higher, then you might experience request rejections and cluster performance issues. To troubleshoot this issue, see How do I troubleshoot high CPU utilization on my OpenSearch Service cluster?

Check for uneven shard distribution

Uneven shard distribution can cause some nodes to handle more traffic than others. This imbalance leads to rejections on overloaded nodes. For more information, see How do I rebalance uneven shard distribution in my OpenSearch Service cluster?

To check your shard distribution, run the following command:

GET /_cat/allocation?v

Review the output to identify nodes with a higher shard count. If your cluster distributes shards unevenly, then rebalance your cluster or adjust your index settings to distribute shards more evenly across nodes.

Check whether your active and queue threads are full

If there's a lack of CPUs or high request concurrency, then a queue can fill up and result in an "HTTP 429" error.

To check your active and queue threads for search rejections, run the following command:

GET /_cat/thread_pool/search?v&h=id,name,active,queue,rejected,completed

Note: Replace search with write to check active and queue threads for write rejections.

Reduce search rejections

OpenSearch Service rejects your search request in the following situations:

All active threads are busy and request queues are at maximum capacity.
The number of requests exceeds the bulk queue capacity.
The search thread pool queue has reached its maximum size of 1,000.

Each node's bulk queue can hold from 50 to 200 requests. The exact capacity depends on your OpenSearch Service or Elasticsearch version.

To monitor and reduce search rejections, complete the following steps:

Configure OpenSearch Service logs to capture search slow logs. To set a threshold for slow log generation, run the following command:
```
curl -XPUT http://your-domain-endpoint/index/_settings -d '{"index.search.slowlog.threshold.query.level":"10s"}'
```
Note: Replace your-domain-endpoint with your domain's endpoint and level with the correct log level. To avoid extra overhead, set your slow log threshold high enough. For example, if most of your queries take 11 seconds, then set your slow log threshold to 20 seconds.
Review the search slow logs to identify slow-running queries.
If rejections persist, then upgrade to a larger instance type or add more data nodes.

For information about how to implement search performance strategies for your queries, see Troubleshooting Elasticsearch searches, for beginners on the Elasticsearch website. Also, see Advanced tuning: Finding and fixing slow Elasticsearch queries on the Elasticsearch website.

Reduce write rejections

OpenSearch Service rejects your write request in the following situations:

All active threads are busy and write request queues are at maximum capacity.
The node's heap memory usage exceeds 95% or the circuit breaker trips.
The number of requests exceeds the bulk queue capacity.
Primary shard issues occur or primary shards can't process writes.

To reduce write rejections, take one or more of the following actions:

Upgrade to a larger instance type or add more data nodes if rejections persist.
Improve the indexing performance of your documents so that the write queue doesn't reach capacity.
Tune bulk size based on your workload and performance requirements. For more information, see Tuning your cluster for indexing speed on the OpenSearch website.
Add exponential retry logic in your application logic so that OpenSearch Service automatically retries failed requests.
If you're ingesting data from Logstash, then tune the worker count and bulk size. It's a best practice to set your bulk size between 3 and 5 MB.

Note: If your cluster continuously experiences high concurrent requests, then the exponential retry logic doesn't resolve search or write rejections. Use this best practice for a sudden or occasional spike of traffic.