AWS Builder Center: Learn, Build and Connect with builders in the AWS community
AWS Builder Center is the official home for builders on AWS. Share and read what others are working on, follow people who inspire you, explore training and workshops, and find tools to support what you're building.
Why Multi-Region Replication Does Not Replace Backups — Protecting Databases and Streaming Analytics Pipelines from Ransomware
A common misconception in multi-region architectures: "I have Aurora Global Database replicating to a secondary region — I'm protected from data loss." This is false for one critical threat class: data corruption that propagates through replication. Ransomware, accidental bulk deletes, application-level logic bugs, and malicious insiders all generate valid write operations that replicate faithfully to your secondary region.
How Replication Propagates Corruption
Consider this architecture:
┌─────────────────────┐ Replication (< 1s) ┌─────────────────────┐
│ US-WEST-2 (Primary) │ ─────────────────────────────────→ │ US-EAST-2 (Secondary)│
│ │ │ │
│ Aurora Writer │ │ Aurora Reader │
│ (compromised) │ │ (also corrupted) │
└─────────────────────┘ └─────────────────────┘
Attack timeline:
| Time | Event |
|---|---|
| T+0s | Attacker encrypts/corrupts primary database |
| T+<1s | Aurora Global Database replicates corrupted writes to secondary |
| T+5s | Both regions contain encrypted/corrupted data |
| T+60s | Team discovers the issue |
| T+61s | No clean copy exists anywhere in the replication topology |
This applies equally to:
- Aurora Global Database (async replication, typically <1s lag)
- DynamoDB Global Tables (multi-region, multi-active)
- S3 Cross-Region Replication (objects replicate including overwrites/deletes)
- EBS snapshots copied cross-region (if source is already corrupted)
- FSx replication / SnapMirror (mirrors corruption faithfully)
- Kinesis Data Streams cross-region replication (corrupted records flow downstream to analytics consumers)
What Replication Protects Against vs. What It Doesn't
| Threat | Multi-Region Replication | Independent Backups |
|---|---|---|
| Regional infrastructure failure | ✅ | ✅ |
| AZ impairment | ✅ | ✅ |
| Hardware failure | ✅ | ✅ |
| Ransomware / encryption attack | ❌ | ✅ |
Malicious insider (DELETE FROM) | ❌ | ✅ |
| Application bug writing corrupt data | ❌ | ✅ |
Accidental DROP DATABASE | ❌ | ✅ |
| Compromised admin credentials | ❌ | ✅ (if isolated) |
The Solution: Defense-in-Depth Backup Architecture
Layer 1: Point-in-Time Recovery (PITR)
Enable PITR on all critical databases — this gives you continuous, granular recovery independent of replication:
# Aurora — PITR enabled by default, 1–35 day retention aws rds modify-db-cluster \ --db-cluster-identifier my-cluster \ --backup-retention-period 35 # DynamoDB — enable PITR aws dynamodb update-continuous-backups \ --table-name my-table \ --point-in-time-recovery-specification PointInTimeRecoveryEnabled=true
Why this helps: PITR maintains a transaction log independent of replication. You can restore to any second before the corruption event, even after replication has propagated the damage.
Layer 2: Cross-Account Backup Isolation
Store backups in a separate AWS account that production credentials cannot access:
Production Account (111111111111)
│
│ AWS Backup copy rule
▼
Backup Vault Account (222222222222) ← No production IAM can reach here
│
│ Vault Lock (immutable)
▼
Logically Air-Gapped Vault ← Cannot be deleted even by vault account admin
Configure with AWS Backup:
{ "Rules": [ { "RuleName": "DailyBackupWithCrossAccountCopy", "ScheduleExpression": "cron(0 2 * * ? *)", "TargetBackupVaultName": "production-vault", "Lifecycle": { "DeleteAfterDays": 35 }, "CopyActions": [ { "DestinationBackupVaultArn": "arn:aws:backup:us-east-2:222222222222:backup-vault:isolated-vault", "Lifecycle": { "DeleteAfterDays": 90 } } ] } ] }
Layer 3: Immutable Storage with Vault Lock
AWS Backup Vault Lock enforces a WORM (Write Once Read Many) model — even an administrator with root access cannot delete or modify backups during the retention period:
aws backup put-backup-vault-lock-configuration \ --backup-vault-name isolated-vault \ --min-retention-days 7 \ --max-retention-days 365 \ --changeable-for-days 3
Once the changeable-for-days grace period expires, the lock becomes immutable — it cannot be removed, shortened, or overridden by any principal, including the root account.
Layer 4: Logically Air-Gapped Vault (Highest Protection)
For critical workloads, use AWS Backup's logically air-gapped vault — backups are locked by default, encrypted with AWS-owned keys, and isolated from your organizational control plane:
- Cannot be deleted by any identity in the source account
- Supports cross-account sharing for restore without granting delete permissions
- Single-action cross-Region database snapshot copy (Aurora, Neptune, DocumentDB)
- Multi-party approval for any administrative changes
aws backup create-logically-air-gapped-backup-vault \ --backup-vault-name ransomware-resilient-vault \ --min-retention-days 30 \ --max-retention-days 365
Complete Ransomware-Resilient Architecture
┌─────────────────────────────────────────────────────────────────────────────┐
│ PRODUCTION ACCOUNT │
│ │
│ ┌──────────────┐ replication ┌──────────────┐ │
│ │ Aurora Primary│ ───────────────→ │ Aurora Reader │ ← BOTH AT RISK │
│ │ (us-west-2) │ │ (us-east-2) │ │
│ └──────┬───────┘ └──────────────┘ │
│ │ │
│ │ AWS Backup (PITR + scheduled snapshots) │
│ ▼ │
│ ┌──────────────┐ │
│ │ Local Vault │ ← Still at risk if account compromised │
│ │ (Vault Lock) │ │
│ └──────┬───────┘ │
└─────────┼───────────────────────────────────────────────────────────────────┘
│ Cross-account copy
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ ISOLATED BACKUP ACCOUNT │
│ │
│ ┌────────────────────────────┐ │
│ │ Logically Air-Gapped Vault │ ← IMMUTABLE, isolated from prod IAM │
│ │ • Vault Lock enforced │ │
│ │ • Multi-party approval │ │
│ │ • 90-day minimum retention │ │
│ │ • Cross-Region copy │ │
│ └────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Validating Recovery with AWS Fault Injection Service (FIS)
Having backups is meaningless if you've never tested restoring them under realistic failure conditions. AWS FIS lets you simulate regional failures and validate your entire recovery chain — including your streaming analytics pipelines.
Test 1: Simulate Regional Database Failure
Use FIS to disrupt network connectivity to your primary Aurora cluster and verify:
- PITR restore works from your cross-account vault
- Your application correctly fails over to the secondary region
- Data integrity checks pass after restore
Test 2: Validate Streaming Pipeline Resilience with Kinesis FIS Actions
If your architecture includes streaming analytics (Aurora CDC → Kinesis Data Streams → Glue/EMR/OpenSearch/Redshift), corruption doesn't stop at the database — it flows downstream. Use FIS native Kinesis Data Streams actions to test your pipeline's resilience:
# Create FIS experiment to inject ThrottlingException on Kinesis consumers aws fis create-experiment-template \ --description "Test analytics pipeline backpressure handling" \ --targets '{ "kinesisStream": { "resourceType": "aws:kinesis:stream", "resourceArns": ["arn:aws:kinesis:us-west-2:111111111111:stream/cdc-events-stream"], "selectionMode": "ALL" } }' \ --actions '{ "injectThrottling": { "actionId": "aws:kinesis:inject-api-error", "parameters": { "errorCode": "ProvisionedThroughputExceededException", "percentage": "80", "duration": "PT10M" }, "targets": {"Streams": "kinesisStream"} } }' \ --stop-conditions '[{"source": "none"}]' \ --role-arn arn:aws:iam::111111111111:role/FISRole
What to validate during the experiment:
- Does your Kinesis consumer (Lambda, KDA, Glue Streaming) handle
ProvisionedThroughputExceededExceptionwith proper exponential backoff? - Do records land in a dead-letter queue or get silently dropped?
- After the fault clears, does the pipeline catch up without duplicates or gaps in your downstream Redshift/OpenSearch analytics?
- Monitor
GetRecords.IteratorAgeMilliseconds— if it climbs indefinitely during the fault and never recovers, your pipeline has a data loss risk
Test 3: End-to-End Recovery Drill
Combine both tests into a quarterly recovery drill:
- Inject FIS network disruption to primary region (simulates regional impairment)
- Verify database failover — Aurora promotes secondary, application reconnects
- Verify streaming pipeline recovery — Kinesis consumers resume from checkpoint, no data loss in downstream analytics
- Run Athena validation queries against your Redshift/S3 lakehouse to confirm data completeness post-failover
# After failover, validate data completeness in Redshift via Athena # Compare record counts between primary pipeline output and DR pipeline output SELECT DATE_TRUNC('hour', event_timestamp) AS hour, COUNT(*) AS record_count FROM analytics_db.events WHERE event_timestamp > CURRENT_DATE - INTERVAL '1' DAY GROUP BY 1 ORDER BY 1;
Summary: The Minimum Viable Ransomware-Resilient Backup Strategy
| Requirement | Implementation | Why |
|---|---|---|
| Independent from replication | AWS Backup scheduled snapshots + PITR | Replication propagates corruption |
| Immutable | Vault Lock with compliance mode | Prevents attacker/admin from deleting backups |
| Isolated | Cross-account copy to separate account | Compromised prod credentials can't reach backups |
| Air-gapped | Logically air-gapped vault | Highest bar — even account admin can't delete |
| Tested | FIS experiments + quarterly restore drills | Untested backups are Schrödinger's backups |
| Streaming validated | FIS Kinesis actions + pipeline catch-up verification | Corruption propagates beyond the DB into analytics |
| Monitored | AWS Backup Audit Manager | Prove compliance, detect gaps before an incident |
Key Takeaway
Multi-region replication protects against infrastructure failures — the physical plane. Immutable, isolated backups protect against logical corruption — the data plane. You need both. A ransomware attacker who gains write access to your primary database has, by design, write access to every replica — and every downstream streaming consumer (Kinesis, MSK, OpenSearch) faithfully processes the corrupted data. Only a backup that is architecturally separated from the write path survives, and only FIS testing proves your recovery actually works.
References
- AWS Backup Vault Lock documentation
- Building cyber resiliency with AWS Backup logically air-gapped vault
- AWS Prescriptive Guidance — Safeguard backups using immutable storage
- Aurora Global Database disaster recovery
- Implementing Multi-party approval workflows for AWS Backup
- Perform resilience testing with AWS FIS — Kinesis Data Streams
- Amazon Kinesis Data Streams FIS actions for API errors
- What is AWS Fault Injection Service?
- Language
- English
Well written and accurate... This is an important topic in this day and age. One small piece of information I would add is that backup retention periods can be different in different regions which may have a small benefit in cost reduction.
replied a month ago
Relevant content
asked 8 months ago
