How do I troubleshoot unassigned shards in my OpenSearch Service cluster?

Short description

You might have unassigned shards in your OpenSearch Service cluster for the following reasons:

• Failed cluster nodes
• Misconfigured replica count
• Shard failed to get an in-memory lock
• Shard limit exceeded
• Disk space issues
• Disk usage skewed and shard allocation strategy
• ClusterBlockException error

Resolution

Identify the reason for your unassigned shards

To identify the unassigned shards and get additional details, complete the following steps:

1. To list the unassigned shards, run the following command:

   curl -XGET 'domain-endpoint/_cat/shards?h=index,shard,prirep,state,unassigned.reason' | grep UNASSIGNED

   Note: If you use AWS Identity and Access Management (IAM) credentials or database credentials with fine-grained access control (FGAC) activated, then you must complete additional steps. Make sure that you sign requests to the OpenSearch Service APIs with your IAM or database user credentials.

2. To retrieve details about your unassigned shards, run the following command:

   curl -XGET 'domain-endpoint/_cluster/allocation/explain?pretty'

3. (Optional) If you use the Kibana or OpenSearch Dashboards console, then run the following command:

   GET _cluster/allocation/explain

Check for failed cluster nodes

If a failed cluster node occurs, then you might have unassigned shards. High CPU usage on the cluster or hardware failures might cause failed cluster nodes. To determine whether the cluster is overloaded, check the CPUUtilization and JVMMemoryPressure metrics. If the cluster is overloaded, then reduce traffic to the cluster. For instructions, see How do I troubleshoot high JVM memory pressure on my OpenSearch Service cluster?

When the node is up, OpenSearch Service automatically re-assigns the shards. If the cluster remains in a red or yellow status, then you might receive an error similar to the following:

"unassigned_info" : {
  "reason" : "MANUAL_ALLOCATION",
  "at" : "2022-03-18T02:45:42.730Z",
  "details" : "failed shard on node [xxxxxxxxxx]: shard failure, reason [corrupt file (source: [flush])], failure FlushFailedEngineException[Flush failed]; nested: CorruptIndexException[codec footer mismatch (file truncated?): actual footer=0 vs expected footer=-1071082520",
  "last_allocation_status" : "no_valid_shard_copy"
},
"can_allocate" : "no_valid_shard_copy",
"allocate_explanation" : "cannot allocate because all found copies of the shard are either stale or corrupt"

To use snapshots to delete and restore these indices, complete the following steps:

1. To identify and delete the red index, run the following commands:

   GET _cat/indices?health=red
   DELETE /index-name

   Note: Replace index-name with the name of your red index.
2. To check for successful snapshots in the automated snapshots repository, run the following command:

   GET _cat/snapshots/Repository-Name?v

   Note: Replace Repository-Name with the name of your repository. OpenSearch Service stores most automated snapshots in the cs-automated repository. If your domain encrypts data at rest, then OpenSearch Service stores them in the cs-automated-enc repository.
3. To restore an index from a snapshot, run the following command:

   POST _snapshot/Repository-Name/Snapshot-ID/_restore
   {
     "indices": "index-name"
   }

   Note: Replace Repository-Name with the name of your repository, Snapshot-ID with your snapshot ID, and index-name with the name of the index you want to restore.

For more information, see Restoring data from snapshots.

Fix misconfigured replica count

If the number of replica shards is greater than the number of data nodes, then you have unassigned shards. OpenSearch Service can't allocate the primary shard and replica shard on the same node.

To resolve this issue, increase the number of nodes. Or, use one of the following commands to reduce the replica count:

Note: Replace index-name with the name of your index and n with the number of replicas you want.

curl -XPUT 'domain-endpoint/index-name/_settings' -H 'Content-Type: application/json' -d '{"index":{"number_of_replicas":n}}'

PUT index-name/_settings
{
    "index" : {
      "number_of_replicas" : n
    }
}

Note: A single node cluster with replica shards always initializes with yellow cluster status. This occurs because no other available nodes exist for OpenSearch Service to assign a replica shard.

Resolve shard in-memory lock failure

If the shard didn't get an in-memory lock for shard allocation, then you receive the following error:

"failed_allocation_attempts" : 5,
"details" : "failed shard on node []: failed to create shard, failure IOException[failed to obtain in-memory shard lock]; nested: ShardLockObtainFailedException[[][5]: obtaining shard lock timed out after 5000ms]; ",
"explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[], failed_attempts[5], delayed=false, details[failed shard on node [xxxxxxxxxxx]: failed to create shard, failure IOException[failed to obtain in-memory shard lock]; nested: ShardLockObtainFailedException[[xxxxxxxxx][5]: obtaining shard lock timed out after 5000ms]; ], allocation_status[no_attempt]]]"

To resolve this error, run the following command to increase the maximum retry settings:

PUT /index-name/_settings
{
  "index.allocation.max_retries": 10
}

Note: Replace index-name with the name of your index.

Verify shard limit per node

By default, Elasticsearch 7.x and later, and all versions of OpenSearch up to 2.15, have a limit of 1,000 shards per node. For OpenSearch 2.17 and later, OpenSearch Service supports 1,000 shards for every 16 GB of JVM heap memory. The maximum is 4,000 shards per node. For more information, see Shard count quotas.

If you set shard allocation filters to control how OpenSearch Service allocates shards, then your shards might become unassigned because you don't have enough filtered nodes. To prevent this node shortage, increase your node count.

To change the max_shards_per_node value, run the following command:

PUT /_cluster/settings
{
  "persistent": {
    "cluster.max_shards_per_node": n
  }
}

Note: Replace n with the max shards for each node that you want. Also, make sure that the number of replicas for every primary shard is less than the number of data nodes.

To check default limits and whether you can edit the limit, see Shard count quotas.

For more information, see Cluster settings API on the OpenSearch website.

Check disk space allocation

Disk-based shard allocation might cause shards to become unassigned. For example, if the cluster.routing.allocation.disk.watermark.low metric is set to 50 GB, then the specified amount of disk space must be available for shard allocation. For more information, see Disk-based shard allocation settings on the Elasticsearch website.

To check the current disk-based shard allocation settings, run the following command:

curl -XGET 'domain-endpoint/_cluster/settings?include_defaults=true&flat_settings=true'

To resolve your disk space issues, take the following actions:

• Delete any unwanted indices for yellow and red clusters
• Delete red indices for red clusters
• Scale up the EBS volume
• Add more data nodes

Note: Avoid making any configuration changes to your cluster when it's in a red health status. If you try to reconfigure your domain when it's in a red cluster status, then it could get stuck in a Processing state.

Check disk usage and shard allocation strategy

Disk usage can be heavily skewed for the following reasons:

• Uneven shard sizes in a cluster
• Insufficient disk space on a node
• Incorrect shard allocation strategy

By default, Amazon OpenSearch Service has a shard allocation strategy of 5:1, where it divides each index into five primary shards. Within each index, each primary shard also has its own replica. OpenSearch Service automatically assigns primary and replica shards to separate data nodes, and makes sure that there's a backup in case of failure.

To rebalance the shard allocation in your OpenSearch Service cluster and update your strategy, see How do I rebalance the uneven shard distribution in my OpenSearch Service cluster?

Resolve ClusterBlockException error

If you try to create an index or write data to your OpenSearch Service domain, then you might receive the ClusterBlockException error. The error is similar to the following:

"reason": "blocked by: [FORBIDDEN/6/cluster read-only (api)];",
"type": "cluster_block_exception"

To resolve this error, see How do I resolve the 403 "index_create_block_exception" or "cluster_block_exception" error in OpenSearch Service?

Related information

Troubleshooting Amazon OpenSearch Service

Choosing the number of shards