Service software update R20260720 deleted all indices and FGAC internal users on a single-node OpenSearch 2.19 domain

Yes, data and credential loss during service software updates is a known risk for single‑node OpenSearch Service domains without replicas. Blue/green deployments can fail or roll back security indices, especially on small t3 nodes. Automated snapshots remain restorable, but you must configure multi‑node domains with replicas and schedule updates during off‑peak windows to ensure safety.

https://repost.aws/questions/QUlDZQFcEJRBm2SOkO0e7tcw/opensearch-domain-silently-loses-all-fgac-security-state-master-user-included

https://repost.aws/knowledge-center/opensearch-software-update-issues

https://docs.aws.amazon.com/opensearch-service/latest/developerguide/release-notes.html

Glad you got back online. Your recovery path was correct and complete.

The root cause here is a structural constraint worth understanding before the next update cycle. On a single-node cluster, OpenSearch physically cannot place a replica shard on the same node as its primary. So with one node, replica_count is always effectively 0 regardless of what you configure. That means every service software update's blue/green migration runs with zero data redundancy during the transition window. AWS documents this explicitly: if the node crashes or the migration hits an anomaly, there is data loss. The .opendistro_security index going with it is the same mechanism, which is why all your FGAC internal users disappeared at the same time.

Three things worth doing before the next required update:

Register a customer-owned S3 bucket as a manual snapshot repository and set up a Snapshot Management policy. The cs-automated snapshots are your recovery floor but you cannot independently verify or trigger them. A customer-controlled repo lets you take an explicit snapshot before any update and confirm it restored cleanly in a test domain.

Add at least one more data node and set replica_count=1 on your application indices. Even two nodes lets OpenSearch distribute primary and replica shards, which is the actual gap that made this incident total rather than partial.

Configure an off-peak update window via the domain's "Auto-Tune" and update settings so future required updates don't land mid-day. You can't stop required updates but you can control when they run.

The t3.small instance type is also worth revisiting for production. AWS's own fault-tolerance guidance calls it out as not suitable for high-availability deployments.

I do AWS architecture and Well-Architected advisory work, so if a more structured resilience review with a full remediation plan would be useful, happy to help.