- Newest
- Most votes
- Most comments
When an OpenSearch Service domain gets stuck during a configuration change like an instance type update, it typically enters a 'Processing' state while creating a new environment with the new nodes. The domain can become stuck if the new data nodes fail to launch properly, shard migration to the new nodes is unsuccessful, or a validation check fails with errors.
In your case, the blue/green deployment appears to have successfully created the new m6g.large nodes and migrated the data (since your cluster is green and the old t3.small node holds no shards), but the final cleanup stage—removing the old resources—hasn't completed. The fact that the old node appears to be recycling rather than being terminated suggests the service is encountering an issue during the cleanup phase.
Unfortunately, when a domain is stuck in this state for an extended period (22 hours in your case), and you're unable to make further configuration changes due to the 'A change/update is in progress' error, this typically requires intervention from AWS Support to resolve. They can investigate why the cleanup stage hasn't completed and help unblock the domain so you can proceed with additional changes.
Without a support plan, your options are limited. You can add correspondence about this issue through the AWS Support Center at https://console.aws.amazon.com/support/. Alternatively, you might consider reviewing your AWS Support plan options, as higher-tier plans provide faster response times for situations like this where a domain is stuck in a configuration change state.
Sources
Troubleshooting Amazon OpenSearch Service - Amazon OpenSearch Service
Instance type changes always trigger a blue/green deployment, which means the cluster node count doubles while data migrates to the new nodes. There's no fixed time limit for how long that takes, the AWS docs are explicit that duration depends on cluster size, shard count, shard size, and current workload, so 22 hours isn't automatically abnormal for a large or heavily sharded domain.
A few things worth checking right now while you wait:
Run the cat recovery API against your domain endpoint to see whether shard relocation is actually making forward progress, even slowly. Something like: curl -X GET "" | awk '/peer/ {print $1" "$2" "$3" "$4" "$18}' | grep -v 100.0%. If percentages are ticking up, the deployment is working and just needs more time. If they've been frozen for hours, that's a different signal.
Also check: JVM memory pressure and CPU on the domain's CloudWatch metrics, consistently high values on either are a documented reason blue/green deployments stall. Same for available disk space, since the migration needs headroom to copy shards before dropping old nodes.
If you're past 24 hours and the cat recovery output shows zero progress, the AWS guidance is to open a Support case at that point, that's the documented threshold for escalating a stuck Modifying state.
One thing that can't be done mid-flight: you can't cancel or pause a blue/green in progress. You can only wait for it to finish, then revert if needed.
I do AWS architecture and operations advisory work and am happy to look at this further if you can share what your cat recovery output is showing.
answered 2 months ago
Relevant content
asked 5 months ago
