How do I resolve the manual snapshot error in my OpenSearch Service cluster?

Resolution

If you don't complete the correct migration process for your OpenSearch Service version, then you receive an error. To troubleshoot the issue, complete the resolution for the error that you receive.

403 Unauthorized error

If you activated fine-grained access control (FGAC) on your OpenSearch Service domain, then you might receive an error message that's similar to the following one when you take a snapshot:

"type: security_exception,
reason: no permissions for [cluster:admin/repository/put] and User [name=arn:aws:iam::012345678912:user/username, backend_roles=[], requestedTenant=null]
status: 403"

The preceding error occurs when you don't provide the correct credentials.

To resolve this issue, run the following curl command to specify a username:password parameter when you take a manual snapshot:

curl -XPUT -u 'username:password' 'opensearch-domain-endpoint/_snapshot/snapshot-repository-name/snapshot-name'

Note: To activate FGAC for your OpenSearch Service domain, you must be a superuser. You can either use your superuser name and password or set an AWS Identity and Access Management (IAM) role as the superuser. When you access your cluster snapshot, specify your superuser credentials or IAM role. If you specify an IAM role, then the IAM role must use AWS Signature Version 4 to sign the HTTP requests. For more information about FGAC and IAM roles, see Creating and managing OpenSearch Service domains.

You must also map the manage_snapshots role to an IAM role, and register a snapshot repository. The manage_snapshots role must have IAM:PassRole permissions to assume the IAM role. For more information, see Prerequisites.

illegal_state_exception

If you use an Amazon Simple Storage Service (Amazon S3) bucket to take manual snapshots for multiple domains, then you get an error message that's similar to the following one:

"type: illegal_state_exception
reason: Can't get text on a FIELD_NAME at 1:1838
status: 500"

To resolve the preceding error, create a new Amazon S3 bucket and take a manual snapshot, or clear all the data from the existing bucket.

Repository_missing_exception

Before you take a manual index snapshot, you must register a manual snapshot repository with OpenSearch Service. You must also meet the manual snapshot prerequisites.

If you didn't register your snapshot repository, the repository name is incorrect, or you didn't fulfill the prerequisites, the you receive the following error message:

"type: repository_missing_exception
reason: [snapshot-repository-name] missing
status: 404"

Concurrent_snapshot_execution_exception

If you try to take a snapshot when another snapshot is in progress, then you receive the following error message:

type: concurrent_snapshot_execution_exception
reason: [snapshot-repository-name:snapshot-name] a snapshot is already running"

To check whether another snapshot is in progress, run the following curl command:

curl -XGET 'opensearch-domain-endpoint/_snapshot/_status'

If a snapshot is in progress, then wait for the current snapshot to complete. If your snapshot is stuck, then check your history of hourly snapshots. For more information, see Why can't I delete an index or upgrade my OpenSearch Service cluster?

Snapshot_restore_exception

If you migrate data from an on-premises cluster to an OpenSearch Service domain, then you might receive the following error message:

type: snapshot_restore_exception
reason: [manual-snapshot-repo:my-manual-snapshot1/HPOcIJryTj6a6GJvyP79bw] the snapshot was created with Elasticsearch version [6.8.0] which is higher than the version of this node [6.7.0]
status: 500"

When you take a snapshot of an existing cluster that runs on a different version of Amazon Elasticsearch than OpenSearch Service, the preceding error occurs. To resolve this issue, upgrade your Elasticsearch version. Or, use the remote reindex API operation to migrate your indexes.

If you activated the domain's FGAC and try to restore the indexes from the snapshot, then you might receive the following error message:

"type: security_exception
reason: no permissions for [] and User [name=username, backend_roles=[], requestedTenant=]
status: 403"

To resolve the preceding error, run the following curl command to exclude the security indexes:

curl -XPOST -u 'username:password' "https://opensearch-domain-endpoint/_snapshot/snapshot-repository/snapshot-id/_restore" -H 'Content-Type: application/json' -d'
{  
"indices": "-.opensearch*,-.opendistro*,-.kibana*"
}'

Note: In the preceding command, replace the example values with your values.

a_w_s_security_token_service_exception

If the IAM role that you associated with your manual snapshot doesn't establish a trust relationship for es.amazonaws.com, then you receive the following error message:

"type: repository_exception
reason: [es_01082021_repo] Could not determine repository generation from root blobs
type: i_o_exception
reason: Exception when listing blobs by prefix [index-]
type: a_w_s_security_token_service_exception
reason: a_w_s_security_token_service_exception: User: arn:aws:sts::332315457451:assumed-role/cp-sts-grant-role/swift-us-west-2-prod-679203657591 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::679203657591:role/ES_Backup_Role (Service: AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied; Request ID: 36d09b93-d94f-457e-8fa5-b0a50ba436c3)
status: 500"

To resolve this issue, make sure that you specify the IAM role that's associated with the manual snapshot when you register the snapshot repository. If you don't have an IAM role, then create one.

Also, check that the trust relationship for the IAM role specifies OpenSearch Service in the Principal statement.

Example trust relationship:

{
    "Version": "2012-10-17",
    "Statement": [{
        "Sid": "",
        "Effect": "Allow",
        "Principal": {
            "Service": "es.amazonaws.com"
        },
        "Action": "sts:AssumeRole"
    }]
}

PARTIAL snapshot state

A snapshot enters the PARTIAL state for the following reasons:

• Unavailable primary shards in your cluster
• Red cluster status
• Node crash

If the snapshot is in the PARTIAL state, then OpenSearch Service didn't store data from a shard. You can restore data from a partial snapshot, but you must use earlier snapshots to restore missing indexes. To check whether your cluster is in the PARTIAL state, check your snapshot history.

Amazon S3 Glacier storage class issue

If you store a restored snapshot in one of the Amazon S3 Glacier storage classes, then don't apply an Amazon S3 Glacier Lifecycle rule to the bucket. Manual snapshots don't support the S3 Glacier storage classes. If you apply an S3 Glacier Lifecycle policy to the bucket, then move objects that transition to the bucket back to a standard storage class.

After you move the objects back, you can restore the objects from the snapshots. For more information, see Prerequisites.

Related information

How do I resolve the "cannot restore index [.kibana]" error in Amazon OpenSearch Service?

Taking manual snapshots