Skip to content

OpenSearch Dashboard intermittent yellow/red status after migration to another node type.

0

After migrating i3.xlarge.search->r7g.xlarge.search by blue/green deployment OpenSearch Dashboards started moving to the yellow state every 2-4 hours and the red state every couple of days(2 times so far). The yellow state continues for 2 min, the red state for 30-40 min. The cluster consists of 4 nodes(no master nodes). Nothing in the logs during the yellow state or the red state. The load in the cluster is fine, as well as free disk space. I can't see any spikes or metric correlations except connections that drop right 1 min minute before the Dashboard moves to the yellow state. I tried to:

  1. Increase throughput and IOPS for EBS storage.
  2. Restart Dashboard/Kibana process on every node. Opensearch dashboard yellow/red status

asked 2 months ago112 views

3 Answers
0

The yellow/red status after a node type migration is almost always unassigned replica shards. When you change instance types, the cluster has to re-rack shards across the new nodes and if anything trips up during that reshuffle, replicas end up unallocated and the cluster turns yellow or red.

Two things worth running right away. First, hit the cluster allocation explain API: GET _cluster/allocation/explain. That gives you the exact reason each shard is stuck, which cuts the guessing. Second, check GET /_cat/indices?v&health=yellow to pinpoint which indices are affected.

One thing to watch: you should resolve the health issue before attempting any further domain reconfigurations, because reconfiguring a domain already in red can get it stuck in "Modifying" indefinitely. So don't retry the instance change until shards are green.

If the allocation explain output shows something like "no nodes have enough disk" or a zone awareness mismatch, that narrows things further. Happy to dig into whatever the API returns with you if you paste the output here.

answered 2 months ago

  • Hi Ajit N K, Thanks for your attention to the question. Unfortunately I couldn't find any unassigned shards. GET _cluster/allocation/explain { "error": { "root_cause": [ { "type": "illegal_argument_exception", "reason": "unable to find any unassigned shards to explain [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDecisions?=false]" } ], "type": "illegal_argument_exception", "reason": "unable to find any unassigned shards to explain [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDecisions?=false]" }, "status": 400 }

    GET /_cat/indices?v&health=yellow returns an empty response

    GET _cluster/health?filter_path=status,number_of_nodes,unassigned_shards { "status": "green", "number_of_nodes": 4, "unassigned_shards": 0 }

0

The intermittent yellow and red status you're experiencing after migrating from i3.xlarge.search to r7g.xlarge.search instances could be related to several factors. Since you mentioned the cluster has 4 nodes without dedicated master nodes, this configuration may be contributing to instability. Without dedicated master nodes, the cluster management responsibilities fall on your data nodes, which can lead to resource contention and stability issues, especially during periods of activity.

The connection drops you're observing one minute before the Dashboard moves to yellow state are a significant clue. This suggests potential resource exhaustion or network-related issues. Since you're using r7g instances (which are ARM-based Graviton processors), there could be differences in how resources are allocated compared to the previous i3 instances (which are Intel-based with local NVMe storage).

Given that your cluster health is otherwise good with adequate disk space and normal load, consider these steps:

First, add dedicated master nodes to your cluster. This is a critical best practice for production workloads as it separates cluster management from data operations, significantly improving stability and resilience. Master nodes handle cluster state management and elections, which can prevent the types of intermittent issues you're experiencing.

Second, ensure you have at least three data nodes distributed across multiple Availability Zones for high availability and fault tolerance. This configuration helps maintain cluster stability during node failures or resource constraints.

Third, review your instance type choice. The r7g.xlarge instances are memory-optimized, but if your workload characteristics differ from what these instances are optimized for, you might experience performance issues. Consider whether your workload would benefit from compute-optimized (C series) or other instance families.

Finally, since the blue-green deployment process itself can be resource-intensive and may cause temporary performance impacts, monitor your cluster metrics closely during the stabilization period after migration. The intermittent nature of your issues suggests the cluster may still be adjusting to the new instance types.

If these issues persist after implementing dedicated master nodes and ensuring proper cluster configuration, you may want to review your AWS Support plan options for more detailed assistance with your specific cluster configuration.
Sources
How to resolve OpenSearch cluster performance issues with t2/t3 burstable instances | AWS re:Post
Minimizing Downtime during Amazon OpenSearch Service Cluster Node Type Upgrade: A Blue-Green Deployment with Failover Strategy | AWS re:Post

answered 2 months ago

EXPERT

reviewed 2 months ago

0

Upgraded Dashboard to the latest OpenSearch_2_15_R20260720 version. Spikes keep occurs but now in red status. Enter image description here

answered 2 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.