Minimizing Reindex Failures on Large Amazon OpenSearch Indices: Why Cluster Capacity Must Be Validated First

The Problem

A reindex operation on a production cluster creates a dual-workload scenario:

1. Live search traffic — scroll contexts, field data, query caches, aggregations
2. Bulk indexing — document parsing, indexing buffers, segment merges, translog writes

Both compete for the same JVM heap. If the cluster lacks sufficient headroom, one of two circuit breakers will trip:

• Indexing pressure circuit breaker — rejects bulk requests when indexing memory exceeds thresholds, resulting in rejected_execution_exception on bulk threads
• Parent circuit breaker — terminates the reindex task when total JVM usage exceeds the configured limit (typically ~95% of heap)

In both cases, the reindex fails partway through, leaving a partial destination index that may degrade live search performance until cleaned up.

---

How to Calculate Required Capacity for a Reindex Operation

The Core Question

Before reindexing, answer: "Does my cluster have enough free resources to handle the reindex while continuing to serve live traffic?"

Three resources determine success or failure:

1. JVM Heap — for scroll contexts (reading source) + bulk indexing buffers (writing destination)
2. CPU — for document parsing, segment merges, and concurrent search
3. Disk — source and destination index coexist until cutover

---

JVM Heap Sizing

The reindex operation consumes heap on every data node involved. Estimate with:

Reindex Heap Per Node ≈ (Source Index Size / Number of Data Nodes) × 0.05 to 0.10

This 5–10% factor accounts for scroll buffers, bulk indexing buffers, translog entries, and segment merge overhead.

For k-NN vector indices or deeply nested documents, use 10–15% — these data types consume significantly more heap per document during indexing.

Example:

Source Index Size	Data Nodes	Per-Node Share	Estimated Reindex Heap Needed
100 GB	10 nodes	10 GB/node	0.5 – 1.0 GB per node
500 GB	20 nodes	25 GB/node	1.25 – 2.5 GB per node
1 TB	40 nodes	25 GB/node	1.25 – 2.5 GB per node
1 TB (k-NN)	40 nodes	25 GB/node	2.5 – 3.75 GB per node

Check Available Heap

GET _cat/nodes?v&h=name,heap.max,heap.current,heap.percent

Calculate:

Available Heap Per Node = (Node Heap Size × 0.85) – Current Heap Usage at Peak

Use 85% as the ceiling (not 95%) because the parent circuit breaker trips at ~95% and you need buffer to avoid it.

Decision Matrix: JVM

Available Heap Per Node vs. Reindex Heap Needed	Verdict
Available > 2× Needed	✅ Proceed with confidence
Available = 1–2× Needed	⚠️ Proceed with throttling and active monitoring
Available < Needed	❌ Scale up first — reindex will fail

---

Disk Space Sizing

Both source and destination index exist simultaneously during the reindex:

Required Free Disk = Source Index Primary Store Size × (1 + destination replicas)

If destination replicas = 0 during reindex (recommended):

Required Free Disk = Source Index Primary Store Size

Check it:

GET _cat/indices/my-source-index?v&h=index,pri.store.size
GET _cat/allocation?v&h=node,disk.avail

If any node has less free disk than (primary store size / number of nodes), the reindex may fail with a disk watermark breach.

---

CPU Sizing

Rule of thumb:

If CPU Utilization at peak > 60% → cluster cannot absorb reindex I/O without impacting search latency

Reindexing adds 20–40% CPU overhead (document parsing, merges, translog fsync). If you are already at 60%, adding a reindex pushes you to 80–100% and search latency degrades.

---

Quick Reference: When to Scale Up

Cluster State at Peak	Can You Reindex?	Action
Heap < 50%, CPU < 40%, Disk > 2× index size free	✅ Yes	Execute with standard parameters
Heap 50–70%, CPU 40–60%, Disk > 1× index size free	⚠️ Maybe	Use conservative throttling, monitor, run off-peak
Heap > 70% OR CPU > 60% OR Disk < index size free	❌ No	Scale up, wait for stabilization, then reindex

---

How Much to Scale Up

If you need to scale up, calculate the target:

Option 1: Vertical scaling (increase instance size)

Target Instance Size = Current Instance Size × (Current Heap % / 50%)

Example: At 75% heap on r7g.2xlarge (32 GB heap):

• Target: 32 GB × (75/50) = 48 GB heap needed
• Move to r7g.4xlarge (64 GB heap) — gives ~50% utilization with headroom for reindex

Option 2: Horizontal scaling (add nodes)

Additional Nodes Needed = Total Reindex Heap Needed / Available Heap Per Existing Node

Critical: Wait for the cluster to fully stabilize (green status, no relocating shards) after scaling before initiating the reindex.

---

Reindex Execution: Conservative Parameters for Large Indices

Once capacity is confirmed sufficient, proceed with these settings.

Prepare the Destination Index

PUT my-large-index-v2
{
  "settings": {
    "index": {
      "number_of_shards": 40,
      "number_of_replicas": 0,
      "refresh_interval": "-1"
    }
  }
}

Setting replicas to 0 and disabling refresh during reindex reduces heap and I/O consumption significantly.

Execute the Reindex

POST _reindex?wait_for_completion=false
{
  "source": {
    "index": "my-large-index-v1",
    "size": 3000
  },
  "dest": {
    "index": "my-large-index-v2"
  },
  "conflicts": "proceed"
}

Parameter guidance:

Parameter	Recommendation	Rationale
wait_for_completion	false	Prevents HTTP timeout on long-running operations
size (batch)	1000–5000	Smaller batches = less heap per scroll
requests_per_second	Start at 2000–3000	Throttles bulk rate; can be adjusted mid-flight
Replicas on destination	0 during reindex	Eliminates duplicate indexing cost; restore after
Refresh interval	-1 during reindex	Prevents segment creation during writes; restore after

---

Monitoring During Execution

Key Metrics to Watch

Metric	Where to Check	Danger Level	Action
JVM Heap %	CloudWatch: JVMMemoryPressure	> 85%	Rethrottle immediately
CPU Utilization	CloudWatch: CPUUtilization	> 90% sustained	Rethrottle immediately
Write Rejections	GET _nodes/stats/thread_pool/write	Any rejections > 0	Reduce requests_per_second
Search Latency	CloudWatch: SearchLatency	> 2× baseline	Rethrottle or pause
Indexing Rate	CloudWatch: IndexingRate	Dropping to 0	Check task status — may have failed

Check Reindex Progress

GET _tasks/<task_id>

Rethrottle Mid-Flight (if metrics exceed thresholds)

POST _reindex/<task_id>/_rethrottle?requests_per_second=500

Cancel If Necessary

POST _tasks/<task_id>/_cancel

After cancellation, delete the partial destination index to stop it from degrading search:

DELETE my-large-index-v2

---

Post-Reindex Steps

Restore Settings

PUT my-large-index-v2/_settings
{
  "index": {
    "number_of_replicas": 1,
    "refresh_interval": "30s"
  }
}

Verify Document Count

GET my-large-index-v1/_count
GET my-large-index-v2/_count

Force Merge (optional, for read-heavy indices)

POST my-large-index-v2/_forcemerge?max_num_segments=1

Scale Back Down (if you scaled up temporarily)

After successful reindex and verification, you can scale the cluster back to the original instance size. Wait for all shards to stabilize before reducing capacity.

---

Pre-Reindex Capacity Checklist

□ JVM heap usage below 60% at peak traffic
□ CPU utilization below 60% at peak traffic
□ Free disk space ≥ source index primary store size
□ Zero write/search thread pool rejections
□ If any check fails → scale up FIRST, stabilize, then reindex
□ Destination index created with replicas=0, refresh_interval=-1
□ Reindex running async (wait_for_completion=false)
□ Monitoring JVM, CPU, and rejections during execution
□ Rethrottle plan ready if metrics exceed thresholds
□ Replicas and refresh restored after completion
□ Document count verified (source vs. destination)
□ Scale back down after successful completion (if temporary scale-up)

---

Key Takeaway

A reindex is a capacity planning event, not just an API call. The question is not how to call the reindex API — it is whether your cluster has sufficient resources to handle it alongside live traffic. Validate capacity first, scale up if needed, then execute. This approach prevents circuit breaker failures, avoids partial index corruption, and eliminates the need for multiple retry attempts that erode customer confidence.

---

Related Resources

• Reindex data in Amazon OpenSearch Service
• How to Reduce Shard Size for Large Indices (>1TB) in Amazon OpenSearch – Reindexing Guide
• Should You Reindex Oversized Shards in Amazon OpenSearch Service? A Decision Framework
• Troubleshoot a circuit breaker exception in OpenSearch Service
• Troubleshoot high JVM memory pressure in OpenSearch Service
• Best practices for right-sizing Amazon OpenSearch Service domains
• Sizing Amazon OpenSearch Service domains
• Monitoring OpenSearch cluster metrics with Amazon CloudWatch
• Circuit breakers in OpenSearch
• OpenSearch Reindex API reference
• Tasks API reference