OpenSearch Engine Version Upgrade Stuck at 40% - Domain Not Responding

Regarding the question and the re:Post Agent’s answer: An OpenSearch Service upgrade stuck at 40% typically indicates a stalled data migration phase during the blue/green deployment. The drop in search threads and domain unresponsiveness suggest the cluster health has severely degraded, halting the shard relocation process.

In-progress upgrades cannot be manually canceled by the user. Since the cluster is unresponsive, standard API diagnostic commands (like _cat/recovery) will fail or time out. If the automated rollback mechanism fails to trigger, AWS Support intervention is strictly required to manually abort the deployment and restore the original cluster state. Opening a support ticket was the correct and only actionable step.

Official documentation on the upgrade process and rollbacks: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/version-migration.html

When an OpenSearch Service domain upgrade gets stuck at a certain percentage, it's typically experiencing issues during the blue/green deployment process that OpenSearch Service uses for upgrades. Based on the symptoms you're describing—particularly the ThreadpoolIndexSearcherThreads and ConcurrentSearchRate dropping to zero—it appears your domain may be stuck during the shard relocation phase of the upgrade.

Unfortunately, once an upgrade is initiated, it cannot be canceled. You'll need to wait for AWS to resolve the underlying issue before the process can complete. Since you've already opened a support case, that's the correct path forward.

Regarding what might be causing the issue, domains commonly get stuck during shard relocation due to several factors: large shard sizes, high index or search load during the upgrade, insufficient disk space, or reaching shard count quotas. The elevated CPUWait you're seeing could indicate resource constraints during the migration.

To better understand what's happening, you can monitor specific CloudWatch metrics like FreeStorageSpace, ClusterStatus, and ClusterIndexWritesBlocked. You can also use the _cat/recovery API command to monitor the shard migration process in real-time, which can give you visibility into which shards are being relocated and their progress.

For resolution time, this varies depending on the underlying cause. AWS will need to investigate and address the specific issue preventing your upgrade from completing. Your support case is the primary channel for getting updates and resolution.

For future upgrades, consider these best practices to minimize the risk of getting stuck: reindex larger shards to smaller sizes beforehand, ensure you're using an appropriate number of shards for your data volume, initiate upgrades during periods of low cluster usage, test updates on non-production domains first, and ensure your cluster has sufficient resources to handle the increased load during an upgrade.
Sources
Troubleshoot stuck or failed upgrades in OpenSearch Service | AWS re:Post
Troubleshooting Amazon OpenSearch Service - Amazon OpenSearch Service