- Newest
- Most votes
- Most comments
The yellow/red status after a node type migration is almost always unassigned replica shards. When you change instance types, the cluster has to re-rack shards across the new nodes and if anything trips up during that reshuffle, replicas end up unallocated and the cluster turns yellow or red.
Two things worth running right away. First, hit the cluster allocation explain API: GET _cluster/allocation/explain. That gives you the exact reason each shard is stuck, which cuts the guessing. Second, check GET /_cat/indices?v&health=yellow to pinpoint which indices are affected.
One thing to watch: you should resolve the health issue before attempting any further domain reconfigurations, because reconfiguring a domain already in red can get it stuck in "Modifying" indefinitely. So don't retry the instance change until shards are green.
If the allocation explain output shows something like "no nodes have enough disk" or a zone awareness mismatch, that narrows things further. Happy to dig into whatever the API returns with you if you paste the output here.
answered 2 months ago
The intermittent yellow and red status you're experiencing after migrating from i3.xlarge.search to r7g.xlarge.search instances could be related to several factors. Since you mentioned the cluster has 4 nodes without dedicated master nodes, this configuration may be contributing to instability. Without dedicated master nodes, the cluster management responsibilities fall on your data nodes, which can lead to resource contention and stability issues, especially during periods of activity.
The connection drops you're observing one minute before the Dashboard moves to yellow state are a significant clue. This suggests potential resource exhaustion or network-related issues. Since you're using r7g instances (which are ARM-based Graviton processors), there could be differences in how resources are allocated compared to the previous i3 instances (which are Intel-based with local NVMe storage).
Given that your cluster health is otherwise good with adequate disk space and normal load, consider these steps:
First, add dedicated master nodes to your cluster. This is a critical best practice for production workloads as it separates cluster management from data operations, significantly improving stability and resilience. Master nodes handle cluster state management and elections, which can prevent the types of intermittent issues you're experiencing.
Second, ensure you have at least three data nodes distributed across multiple Availability Zones for high availability and fault tolerance. This configuration helps maintain cluster stability during node failures or resource constraints.
Third, review your instance type choice. The r7g.xlarge instances are memory-optimized, but if your workload characteristics differ from what these instances are optimized for, you might experience performance issues. Consider whether your workload would benefit from compute-optimized (C series) or other instance families.
Finally, since the blue-green deployment process itself can be resource-intensive and may cause temporary performance impacts, monitor your cluster metrics closely during the stabilization period after migration. The intermittent nature of your issues suggests the cluster may still be adjusting to the new instance types.
If these issues persist after implementing dedicated master nodes and ensuring proper cluster configuration, you may want to review your AWS Support plan options for more detailed assistance with your specific cluster configuration.
Sources
How to resolve OpenSearch cluster performance issues with t2/t3 burstable instances | AWS re:Post
Minimizing Downtime during Amazon OpenSearch Service Cluster Node Type Upgrade: A Blue-Green Deployment with Failover Strategy | AWS re:Post
Upgraded Dashboard to the latest OpenSearch_2_15_R20260720 version. Spikes keep occurs but now in red status.
answered 2 months ago
Relevant content
asked 2 years ago

Hi Ajit N K, Thanks for your attention to the question. Unfortunately I couldn't find any unassigned shards. GET _cluster/allocation/explain { "error": { "root_cause": [ { "type": "illegal_argument_exception", "reason": "unable to find any unassigned shards to explain [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDecisions?=false]" } ], "type": "illegal_argument_exception", "reason": "unable to find any unassigned shards to explain [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDecisions?=false]" }, "status": 400 }
GET /_cat/indices?v&health=yellow returns an empty response
GET _cluster/health?filter_path=status,number_of_nodes,unassigned_shards { "status": "green", "number_of_nodes": 4, "unassigned_shards": 0 }