OpenSearch Service - How to restore .opendistro_security index

Question

The cluster status was red, so I made a list of unallocated shards with [this reference](https://repost.aws/knowledge-center/opensearch-red-yellow-status), 
The `.kibana_1` and `.opendistro_security` shards were unassigned.

```
curl -XGET {domain-endpoint}/_cat/shards?h=index,shard,prirep,state,unassigned.reason  | grep UNASSIGNED
.opendistro_security   0 p UNASSIGNED NODE_LEFT
.opendistro_security   0 r UNASSIGNED PRIMARY_FAILED
.kibana_1              0 p UNASSIGNED NODE_LEFT
.kibana_1              0 r UNASSIGNED REPLICA_ADDED
```

I was able to delete `.kibana_1` and restore it from the snapshot, but when I tried to delete `.opendistro_security` I got the following error and could not delete it.

```
{
  "error": {
    "root_cause": [
       {
         "type": "security_exception",
         "reason": "no permissions for [] and User [name={admin-name}, backend_roles=[], requestedTenant=null]" 
       }
     ],
     "type": "security_exception",
     "reason": "no permissions for [] and User [name={admin-name}, backend_roles=[], requestedTenant=null]" 
  },
  "status": 403
}
```

How can I restore `.opendistro_security` from a snapshot?

Answer

The ".opendistro_security" index is an internal index and managed by AWS OpenSearch. This means that it cannot be deleted from customers end.

In addition, due to special permissions on the OpenSearch Dashboards and fine-grained access control indexes, attempts to restore all indexes might fail, especially if you try to restore from an automated snapshot.

If customer needs to delete/restore the ".opendistro_security" index, they would need to contact our AWS Support team for assistance.

In the meantime, you may want to consider restoring all indexes except the Dashboards and fine-grained access control indexes. You can do this using the following command:

```
curl -XPOST 'domain-endpoint/_snapshot/cs-automated/2020-snapshot/_restore' \
-d '{"indices": "-.kibana*,-.opendistro*"}' \
-H 'Content-Type: application/json'
```

This will restore all indexes in your cluster except for the Dashboards and fine-grained access control indexes.

For more information kindly refer below documentation:
[+] Restoring snapshots in Amazon OpenSearch Service - https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-snapshots.html#managedomains-snapshot-restore

Please consider opening a support case for further assistance if required.

OpenSearch Service - How to restore .opendistro_security index

Relevant content