What does opensearch error type: cluster_block_exception reason: blocked by: [FORBIDDEN/8/index write (api)....... mean?

0

Seeing this cryptic error being generated from our ruby on rails application that uses aws opensearch service:

[403] {"error":{"root_cause":[{"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/8/index write (api)];"}],"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/8/index write (api)];"},"status":403}

What does this mean?

asked a year ago1513 views
1 Answer
0

Hello,

You might receive a ClusterBlockException error for the following reasons:

  1. Lack of available storage space
  2. High JVM memory pressure

[+] Troubleshooting Amazon OpenSearch Service - ClusterBlockException - https://docs.aws.amazon.com/opensearch-service/latest/developerguide/handling-errors.html#troubleshooting-cluster-block

Lack of available storage space

If one or more nodes in your cluster has storage space less than the minimum value of 1) 20% of available storage space, or 2) 20 GB of storage space, basic write operations like adding documents and creating indexes can start to fail.

To avoid issues, monitor the FreeStorageSpace metric in the OpenSearch Service console and create CloudWatch alarms to trigger when FreeStorageSpace drops below a certain threshold. GET /_cat/allocation?v also provides a useful summary of shard allocation and disk usage. To resolve issues associated with a lack of storage space, scale your OpenSearch Service domain to use larger instance types, more instances, or more EBS-based storage.

High JVM memory pressure

When the JVMMemoryPressure metric exceeds 92% for 30 minutes, OpenSearch Service triggers a protection mechanism and blocks all write operations to prevent the cluster from reaching red status. When the protection is on, write operations fail with a ClusterBlockException error, new indexes can't be created, and the IndexCreateBlockException error is thrown.

When the JVMMemoryPressure metric returns to 88% or lower for five minutes, the protection is disabled, and write operations to the cluster are unblocked.

High JVM memory pressure can be caused by spikes in the number of requests to the cluster, unbalanced shard allocations across nodes, too many shards in a cluster, field data or index mapping explosions, or instance types that can't handle incoming loads. It can also be caused by using aggregations, wildcards, or wide time ranges in queries.

To reduce traffic to the cluster and resolve high JVM memory pressure issues, try one or more of the following:

  • Scale the domain so that the maximum heap size per node is 32 GB.
  • Reduce the number of shards by deleting old or unused indexes.
  • Clear the data cache with the POST index-name/_cache/clear?fielddata=true API operation. Note that clearing the cache can disrupt in-progress queries.

In general, to avoid high JVM memory pressure in the future, follow these best practices:

  • Avoid aggregating on text fields, or change the mapping type for your indexes to keyword.
  • Optimize search and indexing requests by choosing the correct number of shards.
  • Set up Index State Management (ISM) policies to regularly remove unused indexes.

[+] https://repost.aws/knowledge-center/opensearch-high-jvm-memory-pressure

AWS
SUPPORT ENGINEER
Rajat_C
answered a year ago
  • We have a 10gig instance. There is 7gig free. We should expand that to at least 20gig?

  • Can we increase the ebs volume size without losing data?

  • Hello,

    We will need to check the Amazon OpenSearch domain to narrow down the exact cause of ClusterBlockException - whether it is Lack of available storage space or High JVM memory pressure before recommending any changes.

    If you continue to experience the issue after performing basic troubleshooting as discussed previously, please reach out to support engineering for further assistance and recommendations.

  • Hello,

    Yes, you can increase EBS volume without losing data. Changing EBS volume will cause a Blue/Green deployment.

    As you might be aware, Amazon OpenSearch uses a blue/green deployment process when updating domains, which follows the below process: → AWS will provision a new set of nodes with the desired configuration. → Once these nodes have finished provisioning correctly, data is migrated from your old nodes to the new ones. → Once this migration process is complete, then the old nodes are terminated, leaving you with the new desired configuration.

    [+] Making configuration changes in Amazon OpenSearch Service - Changes that usually cause blue/green deployments - https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-configuration-changes.html#bg

  • jvm memory pressure is around 90%. Think need to relieve it

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions