Skip to content

OpenSearch domain stuck in ApplyingChanges for 22h after instance type change

0

Hello,

An OpenSearch domain (eu-south-1, engine 3.7) is stuck in ConfigChangeStatus: ApplyingChanges since yesterday 12:54 CEST — 22 hours — after an instance type change. The blue/green deployment is blocked on its last stage, "Deleting older resources", which has been PENDING the whole time. As a result, every UpdateDomainConfig call is rejected with "A change/update is in progress", so I cannot make any further change to the domain. The cluster itself is healthy: green, ingestion working. The issue is the stuck change, not availability. Is there a way to unblock this without a support plan?

Instance type changed from t3.small.search to m6g.large.search via Terraform, yesterday at 12:14 CEST. The first four stages completed; the fifth never started. describe-domain reports Processing: false and UpgradeProcessing: false, while ChangeProgressDetails.ConfigChangeStatus is still ApplyingChanges. LastUpdatedTime has not moved since 12:54. The cluster has 3 nodes instead of 2: two m6g.large and one t3.small left from the old environment. The old node holds 0 shards and 0 bytes. The old node appears to be recycled rather than removed: yesterday evening it had 8.3h uptime, this morning a different node id with 4h uptime, same 1GB heap. Six ENIs exist for the domain: three in-use, three available. Unchanged for 22 hours. The same change was applied to a second domain in the same account and region, same engine version, same instance types. There the blue/green completed normally in about 45 minutes.

regards,

asked 2 months ago101 views

2 Answers
0

When an OpenSearch Service domain gets stuck during a configuration change like an instance type update, it typically enters a 'Processing' state while creating a new environment with the new nodes. The domain can become stuck if the new data nodes fail to launch properly, shard migration to the new nodes is unsuccessful, or a validation check fails with errors.

In your case, the blue/green deployment appears to have successfully created the new m6g.large nodes and migrated the data (since your cluster is green and the old t3.small node holds no shards), but the final cleanup stage—removing the old resources—hasn't completed. The fact that the old node appears to be recycling rather than being terminated suggests the service is encountering an issue during the cleanup phase.

Unfortunately, when a domain is stuck in this state for an extended period (22 hours in your case), and you're unable to make further configuration changes due to the 'A change/update is in progress' error, this typically requires intervention from AWS Support to resolve. They can investigate why the cleanup stage hasn't completed and help unblock the domain so you can proceed with additional changes.

Without a support plan, your options are limited. You can add correspondence about this issue through the AWS Support Center at https://console.aws.amazon.com/support/. Alternatively, you might consider reviewing your AWS Support plan options, as higher-tier plans provide faster response times for situations like this where a domain is stuck in a configuration change state.
Sources
Troubleshooting Amazon OpenSearch Service - Amazon OpenSearch Service

answered 2 months ago

EXPERT

reviewed 2 months ago

0

Instance type changes always trigger a blue/green deployment, which means the cluster node count doubles while data migrates to the new nodes. There's no fixed time limit for how long that takes, the AWS docs are explicit that duration depends on cluster size, shard count, shard size, and current workload, so 22 hours isn't automatically abnormal for a large or heavily sharded domain.

A few things worth checking right now while you wait:

Run the cat recovery API against your domain endpoint to see whether shard relocation is actually making forward progress, even slowly. Something like: curl -X GET "" | awk '/peer/ {print $1" "$2" "$3" "$4" "$18}' | grep -v 100.0%. If percentages are ticking up, the deployment is working and just needs more time. If they've been frozen for hours, that's a different signal.

Also check: JVM memory pressure and CPU on the domain's CloudWatch metrics, consistently high values on either are a documented reason blue/green deployments stall. Same for available disk space, since the migration needs headroom to copy shards before dropping old nodes.

If you're past 24 hours and the cat recovery output shows zero progress, the AWS guidance is to open a Support case at that point, that's the documented threshold for escalating a stuck Modifying state.

One thing that can't be done mid-flight: you can't cancel or pause a blue/green in progress. You can only wait for it to finish, then revert if needed.

I do AWS architecture and operations advisory work and am happy to look at this further if you can share what your cat recovery output is showing.

answered 2 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.