How do I resolve low storage space in my OpenSearch Service domain?

Resolution

Your OpenSearch Service domain has storage space requirements that you must adhere to. If you run out of domain storage space, then you might receive the "ClusterBlockException" error.

To resolve low storage space issues, take one or more of the following actions.

Check for unbalanced shards

Unbalanced shards (disk space skew) can cause low storage space for some nodes. To resolve this issue, distribute shards evenly across all nodes.

To check how much storage space is available for each node in your cluster, run the following command:

curl -XGET "aos_endpoint/_cat/allocation?v"

Note: Replace aos_endpoint with the endpoint that's listed in the OpenSearch Service console.

Example output:

shards | disk.indices | disk.used | disk.avail | disk.total | disk.percent | host         | ip          | node
8    |   989.4kb    |   25.9gb    |   32.4gb   |   58.4gb   |   44         | 192.0.2.0    | 192.0.2.3   | node1
8    |   962.4kb    |   25.9gb    |   32.4gb   |   58.4gb   |   44         | 192.0.2.1    | 192.0.2.4   | node2

In the output, you can view the disk space metrics for each node. OpenSearch Service reserves either 20 GiB or 20% of the available storage on each Amazon Elastic Compute Cloud (Amazon EC2) instance for internal operations. The preceding output for cat/allocation doesn't include the reserved storage. To view reserved and total data storage, check the FreeStorageSpace OpenSearch Service metric in Amazon CloudWatch.

Note: Because cat/allocation doesn't include reserved data, its value is always lower than the storage that you see in the OpenSearch Service console.

Increase the size of your domain's Amazon EBS volumes

If your domain uses Amazon Elastic Block Store (Amazon EBS) volumes for storage, then increase the size of the EBS volumes.

If you can't increase the size of the EBS volume, then take one or more of the following actions to scale your cluster:

• Increase the size of the EC2 instances for your data nodes.
• Add more nodes to better distribute the load.

Note: If you use Amazon EC2 I3 instances for data storage, then add nodes to your cluster or scale up your instance type. When you scale your instance, you might cause a blue/green deployment.

Delete unused indexes

To reduce the amount of data that's stored in your domain, delete unused indexes or documents, optimize old indexes, or reduce the domain's replica count. If the cluster is already at maximum disk usage, then you must scale your EBS volumes or nodes.

Note: If you reduce the domain's replica count, then the fault tolerance reduces. It's a best practice to configure at least one replica for each index.

First, create a backup of unwanted indexes to your Amazon Simple Storage Service (Amazon S3) bucket. You can use automated snapshots that OpenSearch Service takes to restore your instance. Then, delete the indexes from your OpenSearch Service cluster to free up disk space.

It's a best practice to also take manual snapshots. Before you create a manual backup, you must make a manual snapshot repository.

To check the creation date of indexes, run the following command:

GET _cat/indices?h=h,s,i,id,p,r,dc,dd,ss,creation.date.string&s=creation.date.string:desc

To delete a single index, run the following command:

DELETE index-name

Note: Replace index-name with the index name.

To delete multiple indexes, run the following command:

DELETE index-pattern

Note: Replace index-pattern with your index pattern.

Use ISM to manage low storage space

Use Index State Management (ISM) to roll over indexes in OpenSearch Service. With ISM, you can define custom management policies to mitigate issues such as low disk space. For example, use an ISM policy to automatically delete indexes based on conditions such as index size. Then, use a rollover operation to move a target to a new index when an existing index meets the defined condition.

The following example ISM policy deletes indexes after 50 minutes:

PUT _plugins/_ism/policies/delete_ism_policy  {
    "policy": {
        "policy_id": "delete_ism_policy",
        "description": "A simple default policy that deletes old unused indexes",
        "last_updated_time": 1658834661281,
        "schema_version": 13,
        "error_notification": null,
        "default_state": "example_hot_state",        
        "states": [
            {
                "name": "example_hot_state",
                "actions": [],
                "transitions": [
                    {
                        "state_name": "delete",
                        "conditions": {
                            "min_index_age": "50m"  
                        }
                    }
                ]
            },
            {
                "name": "delete",                    
                "actions": [
                    {
                        "delete": {}
                    }
                ],
                "transitions": []
            }
        ],
        "ism_template": [
            {
                "index_patterns": [                  
                    "sample*"
                ],
                "priority": 100,                    
                "last_updated_time": 1658834436349
            }
        ]
    }
}

Note: The preceding policy attaches to all indexes for the index pattern that you add to the policy.

To attach an ISM policy to the index, run the following command:

POST _plugins/_ism/add/your-index-*  {
     "policy_id": "your_policy_id"
}

Note: Replace your-index-* with your index or index pattern and your_policy_id with your policy ID.

For more information, see How do I use ISM to manage low storage space in OpenSearch Service?

Use CloudWatch alarms to monitor storage

To monitor the amount of available storage in your cluster, use the FreeStorageSpace metric. To receive notifications when your storage space is low, create a CloudWatch alarm for FreeStorageSpace.

For more information, see Recommended CloudWatch alarms for OpenSearch Service.

Related information

Operational best practices for OpenSearch Service

Why is my OpenSearch Service cluster in a red or yellow status?