How do I troubleshoot stuck or failed upgrades in OpenSearch Service?

Short description

OpenSearch Service uses a blue/green deployment process for domain changes. During this process, issues with shard relocation or failed validation checks can cause the domain to get stuck.

Note: You can't pause or cancel an upgrade after you start it. If the upgrade gets stuck, then you must wait for AWS to resolve the underlying issue before the process completes.

Resolution

Check the state of your domain

To check the state of your domain, run the describe-domain-change-process command:

aws opensearch describe-domain-change-progress --domain-name your_domain_name --region aa-example-1

Note: Replace your_domain_name with the name of your domain and aa-example-1 with your AWS Region.

If you initiated a version upgrade, then run the get-upgrade-status command:

aws opensearch get-upgrade-status --domain-name your_domain_name --region aa-example-1

Note: Replace your_domain_name with the name of your domain and aa-example-1 with your Region.

In the output, if you receive SUCCEEDED_WITH_ISSUES, run the get-upgrade-history command to identify the issue for each entry:

aws opensearch get-upgrade-history --domain-name your_domain_name --max-results 10 --region aa-example-1

Note: Replace your_domain_name with the name of your domain and aa-example-1 with your Region.

The domain is stuck during shard relocation

During the blue/green deployment process, OpenSearch Service copies data from the old fleet to the new one. If the shard relocation process gets stuck, then the domain stays in the Processing or Modifying state.

Note: Don't make further configuration changes while the domain is in the Processing state. Only one change can be in progress at a time.

The domain gets stuck in the Processing or Modifying state for the following reasons:

• Large shard sizes of over 50 GB increase the shard copy time.
• An index or search load causes high resource usage on the cluster.
• There isn't enough disk space to copy shards to the new nodes.
• You reached the shard count quota.
• An unassigned primary shard stops the relocation of a shard and results in a red cluster status.

To troubleshoot these issues use Amazon CloudWatch to monitor the FreeStorageSpace, ClusterStatus, and ClusterIndexWritesBlocked OpenSearch Service metrics for disk space issues or high cluster load.

Based on the issues that you identify, take the following actions:

• If FreeStorageSpace is low, then delete unused indexes.
• If ClusterStatus is Red or Yellow, then see Why is my OpenSearch Service cluster in a red or yellow status?
• If ClusterIndexWritesBlocked is True, then delete unused indexes to free disk space. Or, if the Java Virtual Machine (JVM) memory pressure is high, then reduce traffic to the domain.
  Note: For more information about how to delete indexes, see DeleteIndex or Delete Index API on the OpenSearch website.

To monitor the shard migration process, run the following command:

GET _cat/recovery?active_only=true

The command output shows the active shard recovery status, progress percentage, recovery time, failure status, and data transfer size. Check stage, bytes_percent, and time. The bytes_percent column shows a healthy relocation when an increase between polls occurs.

If shards don't progress or you receive an empty output, then there might be issues with the upgrade process. To troubleshoot upgrade issues, contact AWS Support.

If your shard sizes are over 50 GB, then reindex your data to keep individual shard sizes under 50 GB. For instructions, see Reindex data on the OpenSearch website.

If none of the preceding troubleshooting actions resolve the issue, then contact Support.

The domain fails the upgrade validation checks

During an upgrade, OpenSearch Service validates that your domain configuration is compatible with the new version. If the validation fails, then the domain becomes stuck in the Modifying state.

To resolve this issue, check the domain description for failed activities or validation-related error messages. Complete the troubleshooting steps for your error, and then retry your configuration change.

The domain fails the upgrade process

If the upgrade fails, then OpenSearch Service automatically restores the cluster from the snapshot that it took before the upgrade. No further action is necessary.

It's a best practice to take a manual snapshot before you perform any major version upgrade.

To restore the pre-upgrade snapshot onto a new domain, restore your own manual snapshot, or open a support case . To open a support case, contact Support.

Best practices to prevent upgrade issues

Take the following actions:

• Use the correct number of shards for your configuration, and adhere to shard strategy best practices.
• Initiate your upgrades during periods of low cluster usage.
• Test updates on non-production domains first.
• Make sure that your domain has enough CPU, memory, and disk space for the increased load during an upgrade. Monitor your resource usage before you initiate upgrades.
• Test your configuration changes before you apply them.

Related information

Why is my OpenSearch Service domain stuck in the "Modifying" state?

Upgrading Amazon OpenSearch Service domains