OpenSearch Service Domain Stuck in Upgrade for ~24 Hours - Missing Node and Red primary Status

Question

I'm experiencing a critical issue with my OpenSearch Service. An upgrade process has been ongoing for almost 24 hours, one of my nodes is missing, and the primary node connection status is showing red.
The node was missing before the upgrade was started, but didn't notice before applying configuration changes.

**Domain Details:**

Configuration: 2 data nodes (t3.medium.search)
No dedicated primary nodes (using data nodes for primary role)
Current status: Only 1 node visible, primary connection status red

**Current Situation:**

The upgrade has been running for approximately 24 hours with no apparent progress
The cluster health tab shows a red status
Only 1 out of 2 configured nodes is visible in the console
I cannot restart the OpenSearch process as the upgrade is in progress
The domain is effectively unavailable during this time

**Steps Already Taken:**

Verified the domain configuration (2 data nodes configured)

Checked the instance health tab (only 1 node showing)

Monitored the upgrade progress for 24 hours (no change)

re:Post Agent · Answer

When an OpenSearch Service domain gets stuck in the "Processing" state during an upgrade, it's typically because the service is attempting to create a new environment with a new set of nodes, but is encountering issues. In your case, the situation is particularly concerning because you're already missing one node and have a red primary status.

This stuck state can occur due to several reasons:
- The new set of data nodes is failing to launch
- Shard migration to the new nodes is unsuccessful
- A validation check has failed with errors

The fact that you started the upgrade with an already missing node and red primary status likely compounded the problem. When a cluster is in a red state, it indicates that some primary shards are unavailable, which is a critical issue that can prevent successful upgrades.

Since you cannot restart the OpenSearch process while an upgrade is in progress, and the domain has been stuck for 24 hours, you'll need to take more direct action:

1. Check CloudWatch logs for your domain (under /aws/aes/domains/your-domain-name) to look for specific errors during the upgrade process.

2. Use the AWS CLI to check the upgrade status with: `aws opensearch describe-upgrade --domain-name your-domain-name`

3. If possible, check cluster health, shard allocation, and pending tasks through the OpenSearch API.

Given the severity of your situation (missing node, red status, and stuck upgrade), this is likely a case that requires AWS Support intervention. If you have a support plan, you should open a support case immediately under:
- Service: OpenSearch Service
- Category: Domain Management Issue
- Subcategory: Upgrade stuck or failing

If you don't have a support plan, you might need to consider upgrading to at least Developer Support to get assistance with this issue, as stuck upgrades often require manual intervention from AWS to fix a failed step in the rolling upgrade process.

Important: Do not attempt to force-delete or recreate the domain unless advised by AWS Support, as this could lead to data loss.  
**Sources**  
[Troubleshooting Amazon OpenSearch Service - Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/handling-errors.html)  
[OpenSearch Upgrade Stuck at 60% – Urgent Assistance Required | AWS re:Post](https://repost.aws/questions/QUrsCWpZ0KSCKBVvHAIWqqlA/open-search-upgrade-stuck-at-60-urgent-assistance-required)  
[Amazon OpenSearch Service Stuck at Modifying | AWS re:Post](https://repost.aws/questions/QUweGdCLwqRFmBeDjhvJHlVQ/amazon-opensearch-service-stuck-at-modifying)

OpenSearch Service Domain Stuck in Upgrade for ~24 Hours - Missing Node and Red primary Status

Add your answer

Relevant content