How do I troubleshoot low storage space in my OpenSearch Service domain?

Short description

When you process your workload, your OpenSearch Service domain has storage space requirements. For example, if you allocate storage to a cluster node, then up to 20% of that space (20 GB) is reserved. Also, an operating system (OS), such as Linux, reserved 5% of the file system to support critical processes that the root user performs. If you run out of domain storage space, then you might receive the ClusterBlockException error.

To resolve low storage space issues, complete one or more of the following resolutions.

Resolution

Update your cluster sharding strategy

Distribute shards evenly across all nodes. For more information, see Choosing the number of shards.

Note: Before you update your configuration settings, note the configuration changes that can cause a blue/green deployment. Also, make sure that your dedicated primary node type is the recommended node type.

To check how much storage space is available for each node in your cluster, run the following command:

curl -XGET "es_endpoint/_cat/allocation?v"

For more information, see CAT API on the Elastic website.

To update your OpenSearch Service domain cluster sharding strategy, complete the following steps:

1. Open the OpenSearch Service console.
2. In the navigation pane, under Managed clusters, choose Domains.
3. Select your domain.
4. Update your domain settings.
   Note: If you use an Amazon Elastic Block Store (Amazon EBS) volume for storage, then update your Storage configuration settings. Or, update the number of data nodes.
5. Choose Submit.

Increase the size of your domain's Amazon EBS volumes

If your domain uses Amazon EBS volumes for storage, then increase the size of the EBS volumes. The node's instance type or Elasticsearch version type determines the maximum volume size. For example, 512 GiB is the maximum volume size for Elasticsearch version 1.5.

If you can't increase the size of your volumes, then add cluster nodes or scale up your domain and choose a new Amazon EBS quota.

Note: If you use Amazon Elastic Compute Cloud (Amazon EC2) I3 instances for data storage, then add nodes to your cluster or scale up your instance type.

Delete unused or old indexes

Reduce the amount of data that you store in your domain. You can delete unnecessary files, optimize old indexes, or reduce the domain's replica count.

Note: If you reduce the domain's replica count, then the fault tolerance reduces. It's a best practice to configure at least one replica for each index.

Create a backup of unwanted indexes to your Amazon Simple Storage Service (Amazon S3) bucket. Then, delete the indexes from your OpenSearch Service cluster to free up disk space.

Note: Before you create backups, you must make a manual snapshot repository.

To check the creation date of indexes, run the following command:

GET _cat/indices?h=h,s,i,id,p,r,dc,dd,ss,creation.date.string&s=creation.date.string:desc

To delete a single index, run the following command:

DELETE index-name

To delete multiple indexes, run the following command:

DELETE cwl-index-*

Or, to rotate data in OpenSearch Service, use curator-opensearch 0.0.13 on the Python Package Index website. Or, use Index State Management (ISM) to rotate indexes in OpenSearch Service.

Use ISM to manage low storage space

ISM allows you to automate routine tasks and then apply the tasks to indexes and index patterns in OpenSearch Service. With ISM, you can define custom management policies that help you maintain issues, such as low disk space. For example, you can use a rollover operation and an ISM policy to automate deletion of old indexes based on conditions, such as index size. The rollover operation rolls over a target to a new index when an existing index meets the defined condition.

The following example ISM policy deletes indexes after 50 minutes:

PUT _plugins/_ism/policies/delete_ism_policy  
{
    "policy": {
        "policy_id": "delete_ism_policy",
        "description": "A simple default policy that deletes old unused indexes",
        "last_updated_time": 1658834661281,
        "schema_version": 13,
        "error_notification": null,
        "default_state": "example_hot_state",        
        "states": [
            {
                "name": "example_hot_state",
                "actions": [],
                "transitions": [
                    {
                        "state_name": "delete",
                        "conditions": {
                            "min_index_age": "50m"  
                        }
                    }
                ]
            },
            {
                "name": "delete",                    
                "actions": [
                    {
                        "delete": {}
                    }
                ],
                "transitions": []
            }
        ],
        "ism_template": [
            {
                "index_patterns": [                  
                    "sample*"
                ],
                "priority": 100,                    
                "last_updated_time": 1658834436349
            }
        ]
    }
}

Note: The preceding policy attaches to all indexes for the index pattern that you add to the policy.

To attach the ISM policy to the index, use the following API call:

POST _plugins/_ism/add/your-index-*  
{
     "policy_id": "your_policy_id"
}

Note: Replace your_policy_id with your information.

For more information, see How do I use ISM to manage low storage space in OpenSearch Service?

Use Amazon CloudWatch alarms to monitor storage

To monitor the amount of available storage in your cluster, use the Amazon CloudWatch FreeStorageSpace metric. Configure CloudWatch alarms so that you receive notifications when your storage space is low.

For more information, see Monitoring OpenSearch cluster metrics with CloudWatch.

Related information

Operational best practices for OpenSearch Service

How do I improve the fault tolerance of my OpenSearch Service domain?

Why is my OpenSearch Service cluster in a red or yellow status?