Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

Reducing Your RTO: Best Practices for Optimizing EBS Volume Hydration When Restoring Amazon RDS Snapshots at Enterprise Scale

30 minute read
Content level: Expert
0

When you restore an Amazon RDS snapshot, the instance shows "available" within minutes but the underlying EBS volumes are still lazily loading data from S3. This hidden hydration period causes 10–75× read latency for hours, silently breaking RTO targets. Unlike EC2, RDS doesn't support Fast Snapshot Restore or PRVI. This article provides engine-agnostic pre-warming techniques for PostgreSQL, Oracle, SQL Server, and Db2, plus Step Functions automation to hydrate hundreds of databases in parallel.

Introduction

When disaster strikes or a rollback is needed, every minute counts. Amazon Relational Database Service (Amazon RDS) makes it straightforward to restore a database from a snapshot, the instance reaches an available status within minutes, regardless of whether the database is 10 GB or 10 TB. But there's a catch that surprises many teams: the database isn't truly ready when RDS says it is.

Behind the scenes, the underlying Amazon Elastic Block Store (Amazon EBS) volumes are still loading data from Amazon Simple Storage Service (Amazon S3). This process, known as lazy loading or volume hydration, means that the first time any data block is accessed, it must be fetched from S3 in real time, introducing significant latency. A query that normally completes in 1–2 seconds can take over a minute, or even time out entirely.

For enterprises managing hundreds or thousands of RDS databases across multiple engine types, this creates a gap between "infrastructure available" and "database truly ready for production load," one that directly impacts your Recovery Time Objective (RTO) and can catch workload teams off guard.

This post provides a comprehensive, practical guide to:

Prior art: If you manage Oracle databases on RDS specifically, the AWS Database Blog post Prewarm an Amazon RDS for Oracle database to reduce the impact of lazy loading (October 2023) covers Oracle-specific pre-warming techniques. This article extends that guidance to a multi-engine enterprise context, covering PostgreSQL, Oracle, SQL Server, and Db2, with fleet-scale automation patterns, hydration time estimation formulas, and CloudWatch-based completion detection.

  • Understanding why lazy loading happens and how it differs between managed RDS and self-managed EC2 databases
  • Estimating hydration times to set realistic RTO expectations
  • Accelerating hydration using engine-agnostic pre-warming techniques across PostgreSQL, Oracle, SQL Server, and Db2
  • Building observable, automated hydration workflows at enterprise scale using AWS Step Functions, AWS Lambda, and Amazon CloudWatch
  • Securely managing database credentials across hundreds of instances using the native RDS–AWS Secrets Manager integration

The Challenge: What Happens When You Restore an RDS Snapshot

The Lazy Loading Problem

Amazon RDS uses Amazon EBS as its underlying storage layer. When you take an RDS snapshot (whether manually, via automated backups, or through AWS Backup), the snapshot data is durably stored in Amazon S3. When you restore from that snapshot, RDS provisions new infrastructure and creates new EBS volumes from the snapshot.

Here's what happens during that restore:

  1. RDS provisions the compute and networking. The EC2 instance, security groups, parameter groups, and option groups are configured. This takes a few minutes.
  2. RDS creates new EBS volumes from the snapshot. The volumes become immediately available, but the data blocks are not physically present on the EBS storage infrastructure. They remain in S3.
  3. The instance reaches available status. Clients can connect, and the database accepts reads and writes.
  4. Lazy loading begins. Data blocks are fetched from S3 in two ways:- On-demand: When an application or query accesses a block that hasn't been loaded yet, EBS fetches it from S3 in real time. This first-access fetch adds significant latency (in our testing, single to double-digit milliseconds per block, compounding under concurrent workloads).
  • Background hydration: EBS also runs a background process that proactively pulls blocks from S3, but this process is non-deterministic, based on observed behaviour it averages 5–10 MB/s with peaks up to 40 MB/s. AWS does not publish a guaranteed background hydration rate or SLA for completion time.

Why This Matters for RTO

The available status in RDS is misleading from an RTO perspective. It tells you the infrastructure is ready, not that the data is loaded. For a restore testing or disaster recovery attestation system, the meaningful metric is time to fully performant, which includes the hydration period.

Consider a 1 TB database on gp3 storage at baseline performance (3,000 IOPS / 125 MiB/s). Without active intervention, background hydration alone can take 3 or more hours to complete. During this time:

  • Read-heavy queries can experience elevated latency of 10 ms or more (vs. normal sub-2 ms)
  • Write latency is also elevated, especially on Multi-AZ deployments where synchronous replication hits un-hydrated blocks on the standby
  • Workload teams who aren't AWS experts may assume the database is ready, encounter performance issues, and not understand why

When Does Lazy Loading Occur?

Lazy loading isn't limited to manual snapshot restores. It occurs in any scenario that creates new EBS volumes from a snapshot:

  • Snapshot restore (manual or via AWS Backup)
  • Point-in-time recovery (PITR)
  • Read replica creation (the replica is built from a snapshot)
  • Single-AZ to Multi-AZ conversion (a snapshot is used to build the standby host)

How EC2 and Self-Managed Databases Solve This

If you run a self-managed database on Amazon EC2, you have direct control over the EBS volumes. This means you can use two EBS features that eliminate or accelerate lazy loading:

EBS Fast Snapshot Restore (FSR)

Fast Snapshot Restore lets you enable FSR on a specific EBS snapshot in specific Availability Zones. When you create a volume from an FSR-enabled snapshot in an enabled AZ, the volume is fully initialized at creation with zero lazy loading, full provisioned performance from the first I/O.

How to use it:

  1. Enable FSR on your snapshot in the target AZ via the EC2 console or API
  2. Create a volume from that snapshot (via ec2 create-volume or instance launch)
  3. The volume delivers full performance immediately

Considerations:

EBS Provisioned Rate for Volume Initialization (PRVI)

Launched in May 2025, PRVI lets you specify a guaranteed hydration rate of 100–300 MiB/s when creating a volume from a snapshot. You pass the VolumeInitializationRate parameter in the ec2 create-volume API call, instance launch block device mappings, or launch templates.

How to use it:

aws ec2 create-volume \
  --availability-zone us-east-1a \
  --volume-type gp3 \
  --snapshot-id snap-07f411eed12ef613a \
  --volume-initialization-rate 300

Considerations:

  • Supported for all EBS volume types and all EC2 instance types
  • Charged based on snapshot data size × specified initialization rate
  • A 500 GB snapshot at 300 MiB/s would be fully hydrated in approximately 28 minutes
  • Cumulative quota of 5,000 MiB/s across concurrent volume creation requests per region

Why Neither Works with RDS Today

Amazon RDS is a managed service. When you call restore-db-instance-from-db-snapshot, RDS internally creates the EBS volumes using its own service account. The key constraints are:

  1. The RDS RestoreDBInstanceFromDBSnapshot API does not expose the VolumeInitializationRate parameter. There is no way to pass PRVI settings through the RDS API.
  2. You cannot enable FSR on RDS-managed snapshots. RDS snapshots are owned by the RDS service account, not your AWS account. You cannot access the underlying EBS snapshots to enable FSR on them.
  3. You don't have direct access to the EBS volumes that back your RDS instance, so you can't run fio or dd against them as you would on EC2.

This is why RDS customers must rely on database-engine-level workarounds to force all blocks to be read from S3 to EBS, effectively performing the hydration through the database engine itself.

Note (Updated July 2026): AWS continues to expand PRVI support across managed services. On July 14, 2026, AWS Elastic Disaster Recovery (DRS) announced support for EBS volume initialization rate, preserving and passing through VolumeInitializationRate values during drill and recovery launches. Additionally, the EBS describe-volume-status API now returns an EstimatedTimeToCompleteInSeconds field for PRVI-enabled volumes, providing predictable completion tracking. While RDS does not yet expose PRVI or FSR for managed snapshot restores, the expanding availability of volume initialization rate across other AWS services signals continued investment in this area. I recommend filing a feature request through your AWS Support contact or Technical Account Manager to express interest in this capability for RDS.

Storage note (July 2026): As of July 1, 2026, Amazon RDS no longer supports restoring snapshots to magnetic storage. All restores now target gp3 or io2 Block Express, the formulas and guidance in this article reflect these storage types exclusively.

Estimating Hydration Time: Setting Realistic RTO Expectations

There is no published SLA for EBS volume hydration speed. However, you can estimate hydration time based on your storage configuration using the following formula:

Hydration Time ≈ Snapshot Data Size / (Effective Throughput × 0.65)

Where:

  • Snapshot Data Size = actual data in the snapshot (not the volume size). You can find this via aws ec2 describe-snapshots → FullSnapshotSizeInBytes.
  • Effective Throughput = min(Storage Throughput, Instance EBS Bandwidth), meaning the lesser of your storage throughput and your instance class's EBS bandwidth ceiling.
  • 0.65 factor = empirical adjustment factor based on our testing, accounts for S3 fetch latency overhead on un-hydrated blocks and database engine I/O overhead during pre-warming. In practice, expect 60–70% of theoretical throughput.

Understanding the Throughput Bottleneck

Hydration speed is bounded by the lesser of two ceilings: your storage throughput and your instance class EBS bandwidth. Even if you provision gp3 at 4,000 MiB/s throughput, a db.r6i.4xlarge instance (max EBS bandwidth ~10,000 Mbps = ~1,250 MB/s) will cap your effective throughput at 1,250 MB/s. Both must be high for fastest hydration.

Why gp3 at Baseline Is the Worst Case

gp3 at baseline delivers 3,000 IOPS and 125 MiB/s throughput. This 125 MiB/s is the total bandwidth available for everything, including background hydration from S3, pre-warming reads, and any application I/O. The background hydration process competes for this same bandwidth.

There's an additional factor: for databases under 400 GiB on gp3 (or under 200 GiB for Oracle), RDS uses a single EBS volume. At 400 GiB and above (200 GiB for Oracle), RDS automatically stripes across 4 volumes in RAID 0, which quadruples the baseline to 12,000 IOPS / 500 MiB/s. If your databases are under these thresholds, they're on a single volume with the absolute minimum throughput.

This is why restore testing on gp3 at baseline shows the same duration regardless of database size because the bottleneck is the fixed 125 MiB/s throughput, not the data volume.

Hydration Time Estimates by Configuration

The following table shows realistic hydration times with active pre-warming (not passive background hydration), accounting for the 0.65 efficiency factor:

DB Sizegp3 Baseline (125 MiB/s)gp3 Striped Baseline (500 MiB/s)gp3 Max Provisioned (4,000 MiB/s)io2 High IOPS (4,000+ MiB/s)
100 GB~20 minutes~5 minutes<2 minutes<2 minutes
500 GB~90 minutes~25 minutes~4 minutes~4 minutes
1 TB~3 hours~50 minutes~7 minutes~7 minutes
5 TB~15 hours~4 hours~35 minutes~35 minutes
10 TB~30 hours~8 hours~70 minutes~70 minutes

Recommended Instance Classes for Hydration

The instance class determines the maximum EBS bandwidth available. For fastest hydration, choose an instance class with the highest EBS bandwidth your budget allows:

Instance ClassMax EBS BandwidthUse Case
db.r8g.48xlarge40,000 Mbps (5,000 MB/s)Maximum hydration speed
db.r7i.24xlarge30,000 Mbps (3,750 MB/s)Good balance of speed and cost
db.r6i.16xlarge20,000 Mbps (2,500 MB/s)Practical for most large databases
db.r6i.8xlarge10,000 Mbps (1,250 MB/s)Minimum recommended for large databases
db.r6i.4xlargeUp to 10,000 MbpsBurstable (may not sustain peak)

Important: Instance classes marked "Up to" (e.g., db.r6i.4xlarge at "Up to 10,000 Mbps") use burstable EBS bandwidth. They can sustain the maximum for 30 minutes at least once every 24 hours, but may throttle under sustained load. For hydration workloads that run longer than 30 minutes, choose a non-burstable instance class (typically 8xlarge and above).

Accelerating Hydration: Engine-Agnostic Pre-warming

Since RDS doesn't expose EBS-level hydration controls, you must force all data blocks to be read through the database engine itself. The good news: each engine has a single, schema-agnostic command that reads every block in the database without needing to know the table structure. No per-database customization is required.

PostgreSQL: pg_prewarm Extension

The pg_prewarm extension is available on Amazon RDS for PostgreSQL and reads specified relations into the operating system buffer cache or PostgreSQL shared buffers, forcing all blocks to be fetched from S3.

-- Enable the extension (one-time setup, persists across restarts)
CREATE EXTENSION IF NOT EXISTS pg_prewarm;

-- Pre-warm all user tables and their indexes
DO $$
DECLARE
    r RECORD;
BEGIN
    -- Pre-warm tables
    FOR r IN
        SELECT schemaname || '.' || tablename AS full_name
        FROM pg_tables
        WHERE schemaname NOT IN ('pg_catalog', 'information_schema')
    LOOP
        BEGIN
            PERFORM pg_prewarm(r.full_name);
        EXCEPTION WHEN OTHERS THEN
            RAISE NOTICE 'Could not prewarm %: %', r.full_name, SQLERRM;
        END;
    END LOOP;

    -- Pre-warm indexes
    FOR r IN
        SELECT schemaname || '.' || indexname AS full_name
        FROM pg_indexes
        WHERE schemaname NOT IN ('pg_catalog', 'information_schema')
    LOOP
        BEGIN
            PERFORM pg_prewarm(r.full_name);
        EXCEPTION WHEN OTHERS THEN
            RAISE NOTICE 'Could not prewarm index %: %', r.full_name, SQLERRM;
        END;
    END LOOP;
END $$;

This script works on any PostgreSQL database regardless of schema. The exception handling ensures that if any individual relation fails (e.g., a temporary table that no longer exists), the script continues with the remaining relations.

Oracle: RMAN Physical Validation

RMAN validation is the most efficient approach for Oracle. It operates at the block level, reads every data file in the database, and has the lowest performance overhead compared to alternatives like Data Pump or full table scans.

BEGIN
    rdsadmin.rdsadmin_rman_util.validate_current_dbfile(
        p_validation_type => 'PHYSICAL',
        p_parallel        => 4
    );
END;
/

Key points:

  • p_validation_type => 'PHYSICAL' performs a physical block-level read without logical validation overhead
  • p_parallel => 4 controls the degree of parallelism. Increase this for larger instances with more CPU cores
  • For a 1 TB database on a well-provisioned instance, RMAN validation typically completes in approximately 60 minutes
  • No knowledge of tables, schemas, or data model is required

SQL Server: DBCC CHECKDB

DBCC CHECKDB with the PHYSICAL_ONLY option reads every page in the database, forcing all blocks to be hydrated from S3.

-- Run for each database on the instance
DBCC CHECKDB ('YourDatabaseName') WITH PHYSICAL_ONLY, NO_INFOMSGS;

Key points:

  • PHYSICAL_ONLY skips logical consistency checks, focusing purely on reading all physical pages. This is faster and sufficient for hydration purposes
  • NO_INFOMSGS suppresses informational messages, reducing output noise
  • For instances with multiple databases, run DBCC CHECKDB for each database sequentially or in parallel depending on available IOPS

Db2: INSPECT CHECK DATABASE

For IBM Db2 on RDS, the INSPECT command reads all storage blocks in the database.

CALL SYSPROC.ADMIN_CMD('INSPECT CHECK DATABASE RESULTS KEEP');

Alternatively, you can use RUNSTATS to scan all tables and indexes:

CALL SYSPROC.ADMIN_CMD(
    'RUNSTATS ON TABLE ALL ON ALL COLUMNS AND INDEXES ALL'
);

Securely Managing Credentials at Scale: RDS and AWS Secrets Manager

Running pre-warming scripts across 900 databases requires secure, centralized credential management. The recommended approach uses the native RDS–AWS Secrets Manager integration rather than creating and managing secrets manually.

Enabling the Native Integration

When creating or modifying any RDS instance, you can instruct RDS to generate and manage the master user password in Secrets Manager automatically:

# When creating a new instance
aws rds create-db-instance \
    --db-instance-identifier my-db-001 \
    --engine postgres \
    --db-instance-class db.r6i.4xlarge \
    --allocated-storage 500 \
    --master-username masteradmin \
    --manage-master-user-password

# When modifying an existing instance to enable the integration
aws rds modify-db-instance \
    --db-instance-identifier my-db-001 \
    --manage-master-user-password \
    --apply-immediately

When this is enabled, RDS:

  • Generates a strong master password automatically
  • Stores it in Secrets Manager (secret names use the rds! prefix)
  • Rotates the password every 7 days by default
  • Makes the secret ARN available via describe-db-instances → MasterUserSecret.SecretArn

Important Limitation for Snapshot Restores

The --manage-master-user-password flag on restore-db-instance-from-db-snapshot is currently supported only for RDS for Oracle. For PostgreSQL, SQL Server, and Db2, the master password from the original instance carries over in the snapshot. To enable Secrets Manager management for these engines after restore, run:

aws rds modify-db-instance \
    --db-instance-identifier restored-db-001 \
    --manage-master-user-password \
    --apply-immediately

This means your automation workflow needs to handle two scenarios:

  1. Oracle instances: Enable --manage-master-user-password during the restore itself
  2. PostgreSQL, SQL Server, Db2: Enable it via a modify-db-instance call immediately after restore completes

Discovering Credentials Programmatically

Your Lambda functions should never hardcode or look up credentials by naming convention. Instead, use the RDS API as the source of truth:

import boto3
import json

rds_client = boto3.client('rds')
secrets_client = boto3.client('secretsmanager')

def get_db_credentials(db_instance_id):
    """Retrieve credentials for an RDS instance using the native
    Secrets Manager integration."""

    # Step 1: Get the secret ARN from the RDS instance metadata
    response = rds_client.describe_db_instances(
        DBInstanceIdentifier=db_instance_id
    )
    instance = response['DBInstances'][0]

    secret_arn = instance.get('MasterUserSecret', {}).get('SecretArn')
    engine = instance['Engine']
    endpoint = instance['Endpoint']['Address']
    port = instance['Endpoint']['Port']

    if not secret_arn:
        raise ValueError(
            f"Instance {db_instance_id} does not have Secrets Manager "
            f"integration enabled. Enable it with: "
            f"aws rds modify-db-instance "
            f"--db-instance-identifier {db_instance_id} "
            f"--manage-master-user-password --apply-immediately"
        )

    # Step 2: Retrieve the actual credentials from Secrets Manager
    secret_response = secrets_client.get_secret_value(SecretId=secret_arn)
    credentials = json.loads(secret_response['SecretString'])

    return {
        'host': endpoint,
        'port': port,
        'engine': engine,
        'username': credentials['username'],
        'password': credentials['password']
    }

IAM Policy for the Hydration Lambda

Scope the Lambda execution role to only access RDS-managed secrets (which use the rds! prefix) and the RDS describe API:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ReadRDSManagedSecrets",
            "Effect": "Allow",
            "Action": "secretsmanager:GetSecretValue",
            "Resource": "arn:aws:secretsmanager:*:*:secret:rds!*"
        },
        {
            "Sid": "DescribeRDSInstances",
            "Effect": "Allow",
            "Action": "rds:DescribeDBInstances",
            "Resource": "*"
        },
        {
            "Sid": "DecryptSecrets",
            "Effect": "Allow",
            "Action": "kms:Decrypt",
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "kms:ViaService": "secretsmanager.*.amazonaws.com"
                }
            }
        }
    ]
}

Throttling Considerations at Scale

AWS Secrets Manager supports 10,000 GetSecretValue requests per second**** per region. Even if all 900 Lambda functions fire simultaneously and each calls GetSecretValue once, that's 900 requests, well within the limit. Throttling is not a practical concern at this scale.

That said, adding the AWS Parameters and Secrets Lambda Extension as a Lambda layer is still recommended. It caches secret values locally within the Lambda execution environment with a configurable TTL (up to 300 seconds), which reduces latency on subsequent invocations and avoids unnecessary API calls if the same Lambda instance handles multiple databases.

VPC Networking Requirements

Since the Lambda functions need to connect to RDS instances (which reside in VPCs), the Lambdas must be configured with VPC access. This requires VPC endpoints for:

  • Secrets Manager: com.amazonaws.<region>.secretsmanager
  • RDS API: com.amazonaws.<region>.rds (for describe-db-instances calls)
  • CloudWatch: com.amazonaws.<region>.monitoring (if reporting custom metrics)
  • Step Functions: com.amazonaws.<region>.states (if using callbacks)

Without these endpoints, Lambda functions in a VPC cannot reach these AWS services.

Observability: Knowing When Hydration Is Complete

One of the biggest gaps today is the absence of a native RDS event or Amazon EventBridge notification for "database fully hydrated." The available status only indicates infrastructure readiness. Here's how to build observability around the hydration process.

CloudWatch Metrics to Monitor

MetricDuring HydrationAfter HydrationSignificance
ReadLatency10–75 ms1–2 msPrimary indicator, most reliable signal
ReadIOPSElevatedNormal baselineS3 fetches count as read operations
ReadThroughputShows hydration bandwidthDrops to workload levelIndicates how fast data is being pulled from S3
WriteLatencyElevated (Multi-AZ)NormalSynchronous replication hits un-hydrated standby
DiskQueueDepthSpikesLow and stableHigh values indicate I/O contention from hydration

Automated Hydration Detection

Build a CloudWatch-based detection mechanism that determines when hydration is complete:

import boto3
from datetime import datetime, timedelta

cloudwatch = boto3.client('cloudwatch')

def is_hydration_complete(db_instance_id, threshold_ms=5.0,
                          consecutive_periods=3):
    """Check if ReadLatency has stabilised below threshold,
    indicating hydration is complete.

    Args:
        db_instance_id: RDS DB instance identifier
        threshold_ms: ReadLatency threshold in milliseconds
                      (default 5ms)
        consecutive_periods: Number of consecutive 5-minute periods
                             below threshold to confirm completion
    Returns:
        True if hydration appears complete, False otherwise
    """
    check_window = consecutive_periods * 5  # minutes

    response = cloudwatch.get_metric_statistics(
        Namespace='AWS/RDS',
        MetricName='ReadLatency',
        Dimensions=[{
            'Name': 'DBInstanceIdentifier',
            'Value': db_instance_id
        }],
        StartTime=datetime.utcnow() - timedelta(minutes=check_window),
        EndTime=datetime.utcnow(),
        Period=300,  # 5-minute periods
        Statistics=['Average']
    )

    datapoints = sorted(
        response['Datapoints'],
        key=lambda x: x['Timestamp']
    )

    if len(datapoints) < consecutive_periods:
        return False  # Not enough data yet

    # ReadLatency is reported in seconds in CloudWatch
    threshold_seconds = threshold_ms / 1000.0
    recent = datapoints[-consecutive_periods:]

    return all(dp['Average'] < threshold_seconds for dp in recent)

What to Report to Your Attestation System

For each restored database, capture three timestamps:

  1. Restore initiated: When the restore-db-instance-from-db-snapshot API call was made
  2. Instance available: When the RDS event RDS-EVENT-0043 ("Restored from snapshot") fires
  3. Fully hydrated: When ReadLatency stabilises below threshold (detected by the monitoring function above)

The difference between (1) and (2) is your infrastructure recovery time. The difference between (1) and (3) is your true RTO, which is the time until the database is genuinely ready for production load.

Putting It All Together: Enterprise-Scale Automation

The following architecture orchestrates the entire workflow: from snapshot restore through credential retrieval, pre-warming, hydration monitoring, and attestation reporting, using AWS Step Functions as the central orchestrator.

Architecture Overview

                        ┌──────────────────────────┐
                        │     AWS Backup / RDS      │
                        │     Snapshot Restore      │
                        └────────────┬─────────────┘
                                     │
                                     ▼
                        ┌──────────────────────────┐
                        │   RDS Event via           │
                        │   Amazon EventBridge      │
                        │   (RDS-EVENT-0043)        │
                        └────────────┬─────────────┘
                                     │
                                     ▼
                   ┌─────────────────────────────────────┐
                   │       AWS Step Functions             │
                   │       (Hydration Orchestrator)       │
                   │                                     │
                   │  1. Enable Secrets Manager           │
                   │     (modify-db-instance if needed)   │
                   │                                     │
                   │  2. Temporarily increase IOPS        │
                   │     (modify-db-instance)             │
                   │                                     │
                   │  3. Wait for modifications to apply  │
                   │                                     │
                   │  4. Discover credentials             │
                   │     (describe-db-instances →         │
                   │      MasterUserSecret.SecretArn)     │
                   │                                     │
                   │  5. Invoke Hydration Lambda          │
                   │     (Map state, by engine type,      │
                   │      MaxConcurrency: 200)            │
                   │                                     │
                   │  6. Monitor ReadLatency              │
                   │     (poll every 5 min until stable)  │
                   │                                     │
                   │  7. Scale IOPS back to production    │
                   │     (modify-db-instance, no downtime)│
                   │                                     │
                   │  8. Report to attestation system     │
                   │     (restore time + hydration time)  │
                   └─────────────────────────────────────┘
                                     │
                          ┌──────────┴──────────┐
                          ▼                     ▼
                 ┌─────────────────┐   ┌─────────────────┐
                 │ Hydration Lambda│   │ Monitor Lambda   │
                 │ (4 engine paths)│   │ (CloudWatch poll)│
                 │                 │   │                  │
                 │ • Get secret ARN│   │ • ReadLatency    │
                 │   from RDS API  │   │ • ReadIOPS       │
                 │ • Fetch creds   │   │ • ReadThroughput │
                 │   from Secrets  │   │                  │
                 │   Manager       │   │ Stable for 15min?│
                 │ • Connect to DB │   │ → Report complete│
                 │ • Run pre-warm  │   │                  │
                 └─────────────────┘   └─────────────────┘

Step Functions State Machine Definition (Simplified)

The following is a simplified representation of the Step Functions workflow. In production, you would add error handling, retry logic, and timeout configurations.

{
  "Comment": "RDS Snapshot Hydration Orchestrator",
  "StartAt": "GetRestoredInstanceDetails",
  "States": {
    "GetRestoredInstanceDetails": {
      "Type": "Task",
      "Resource": "arn:aws:states:::aws-sdk:rds:describeDBInstances",
      "Parameters": {
        "DbInstanceIdentifier.$": "$.db_instance_id"
      },
      "ResultPath": "$.instance_details",
      "Next": "CheckSecretsManagerEnabled"
    },
    "CheckSecretsManagerEnabled": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.instance_details.DbInstances[0].MasterUserSecret.SecretArn",
          "IsPresent": true,
          "Next": "TemporarilyIncreaseIOPS"
        }
      ],
      "Default": "EnableSecretsManager"
    },
    "EnableSecretsManager": {
      "Type": "Task",
      "Resource": "arn:aws:states:::aws-sdk:rds:modifyDBInstance",
      "Parameters": {
        "DbInstanceIdentifier.$": "$.db_instance_id",
        "ManageMasterUserPassword": true,
        "ApplyImmediately": true
      },
      "Next": "WaitForSecretsManagerReady",
      "ResultPath": "$.modify_result"
    },
    "WaitForSecretsManagerReady": {
      "Type": "Wait",
      "Seconds": 60,
      "Next": "RefreshInstanceDetails"
    },
    "RefreshInstanceDetails": {
      "Type": "Task",
      "Resource": "arn:aws:states:::aws-sdk:rds:describeDBInstances",
      "Parameters": {
        "DbInstanceIdentifier.$": "$.db_instance_id"
      },
      "ResultPath": "$.instance_details",
      "Next": "TemporarilyIncreaseIOPS"
    },
    "TemporarilyIncreaseIOPS": {
      "Type": "Task",
      "Resource": "arn:aws:states:::aws-sdk:rds:modifyDBInstance",
      "Parameters": {
        "DbInstanceIdentifier.$": "$.db_instance_id",
        "Iops": 64000,
        "StorageThroughput": 4000,
        "ApplyImmediately": true
      },
      "ResultPath": "$.iops_result",
      "Next": "WaitForIOPSModification",
      "Catch": [
        {
          "ErrorEquals": ["States.ALL"],
          "Next": "InvokeHydrationLambda",
          "Comment": "If IOPS change fails (e.g., cooldown), proceed with current IOPS"
        }
      ]
    },
    "WaitForIOPSModification": {
      "Type": "Wait",
      "Seconds": 120,
      "Next": "InvokeHydrationLambda"
    },
    "InvokeHydrationLambda": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:::function:rds-hydration-prewarm",
      "Parameters": {
        "db_instance_id.$": "$.db_instance_id",
        "engine.$": "$.instance_details.DbInstances[0].Engine",
        "endpoint.$": "$.instance_details.DbInstances[0].Endpoint.Address",
        "port.$": "$.instance_details.DbInstances[0].Endpoint.Port",
        "secret_arn.$": "$.instance_details.DbInstances[0].MasterUserSecret.SecretArn"
      },
      "ResultPath": "$.hydration_result",
      "Next": "MonitorHydrationProgress",
      "TimeoutSeconds": 86400
    },
    "MonitorHydrationProgress": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:::function:rds-hydration-monitor",
      "Parameters": {
        "db_instance_id.$": "$.db_instance_id"
      },
      "ResultPath": "$.monitor_result",
      "Next": "IsHydrationComplete"
    },
    "IsHydrationComplete": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.monitor_result.hydration_complete",
          "BooleanEquals": true,
          "Next": "ScaleBackIOPS"
        }
      ],
      "Default": "WaitAndRecheck"
    },
    "WaitAndRecheck": {
      "Type": "Wait",
      "Seconds": 300,
      "Next": "MonitorHydrationProgress"
    },
    "ScaleBackIOPS": {
      "Type": "Task",
      "Resource": "arn:aws:states:::aws-sdk:rds:modifyDBInstance",
      "Parameters": {
        "DbInstanceIdentifier.$": "$.db_instance_id",
        "Iops.$": "$.original_iops",
        "StorageThroughput.$": "$.original_throughput",
        "ApplyImmediately": true
      },
      "ResultPath": "$.scaledown_result",
      "Next": "ReportToAttestation"
    },
    "ReportToAttestation": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:::function:rds-hydration-report",
      "Parameters": {
        "db_instance_id.$": "$.db_instance_id",
        "restore_start_time.$": "$.restore_start_time",
        "available_time.$": "$.available_time",
        "hydration_complete_time.$": "$.monitor_result.completion_time"
      },
      "End": true
    }
  }
}

The Hydration Lambda Function

A single Lambda function handles all four engine types. It receives the engine type, endpoint, and secret ARN as input from Step Functions, fetches credentials from Secrets Manager, connects to the database, and runs the appropriate pre-warming command.

import boto3
import json
import logging
import psycopg2          # PostgreSQL
import oracledb          # Oracle
import pyodbc            # SQL Server
import ibm_db            # Db2

logger = logging.getLogger()
logger.setLevel(logging.INFO)

secrets_client = boto3.client('secretsmanager')


def handler(event, context):
    db_id = event['db_instance_id']
    engine = event['engine']
    endpoint = event['endpoint']
    port = event['port']
    secret_arn = event['secret_arn']

    logger.info(f"Starting hydration for {db_id} (engine={engine})")

    # Retrieve credentials from Secrets Manager
    secret = secrets_client.get_secret_value(SecretId=secret_arn)
    creds = json.loads(secret['SecretString'])
    username = creds['username']
    password = creds['password']

    # Route to engine-specific pre-warming
    if engine in ('postgres', 'aurora-postgresql'):
        result = prewarm_postgresql(endpoint, port, username, password)
    elif engine.startswith('oracle'):
        result = prewarm_oracle(endpoint, port, username, password)
    elif engine.startswith('sqlserver'):
        result = prewarm_sqlserver(endpoint, port, username, password)
    elif engine.startswith('db2'):
        result = prewarm_db2(endpoint, port, username, password)
    else:
        raise ValueError(f"Unsupported engine: {engine}")

    logger.info(f"Hydration complete for {db_id}: {result}")
    return {'status': 'complete', 'engine': engine, 'details': result}


def prewarm_postgresql(host, port, user, password):
    """Pre-warm all user tables and indexes using pg_prewarm."""
    conn = psycopg2.connect(
        host=host, port=port, user=user, password=password,
        dbname='postgres', connect_timeout=30
    )
    conn.autocommit = True
    cur = conn.cursor()

    # Ensure pg_prewarm extension exists
    cur.execute(
        "CREATE EXTENSION IF NOT EXISTS pg_prewarm;"
    )

    # Get all user databases
    cur.execute("""
        SELECT datname FROM pg_database
        WHERE datistemplate = false
          AND datname NOT IN ('rdsadmin', 'template0', 'template1')
    """)
    databases = [row[0] for row in cur.fetchall()]
    cur.close()
    conn.close()

    total_relations = 0
    for dbname in databases:
        try:
            db_conn = psycopg2.connect(
                host=host, port=port, user=user,
                password=password, dbname=dbname,
                connect_timeout=30
            )
            db_conn.autocommit = True
            db_cur = db_conn.cursor()

            db_cur.execute(
                "CREATE EXTENSION IF NOT EXISTS pg_prewarm;"
            )

            # Pre-warm all user tables
            db_cur.execute("""
                SELECT schemaname || '.' || tablename
                FROM pg_tables
                WHERE schemaname NOT IN
                      ('pg_catalog', 'information_schema')
            """)
            for (relation,) in db_cur.fetchall():
                try:
                    db_cur.execute(
                        f"SELECT pg_prewarm('{relation}')"
                    )
                    total_relations += 1
                except Exception as e:
                    logger.warning(
                        f"Could not prewarm {relation} "
                        f"in {dbname}: {e}"
                    )

            # Pre-warm all user indexes
            db_cur.execute("""
                SELECT schemaname || '.' || indexname
                FROM pg_indexes
                WHERE schemaname NOT IN
                      ('pg_catalog', 'information_schema')
            """)
            for (index,) in db_cur.fetchall():
                try:
                    db_cur.execute(
                        f"SELECT pg_prewarm('{index}')"
                    )
                    total_relations += 1
                except Exception as e:
                    logger.warning(
                        f"Could not prewarm index {index} "
                        f"in {dbname}: {e}"
                    )

            db_cur.close()
            db_conn.close()
        except Exception as e:
            logger.error(
                f"Could not connect to database {dbname}: {e}"
            )

    return f"Pre-warmed {total_relations} relations " \
           f"across {len(databases)} databases"


def prewarm_oracle(host, port, user, password):
    """Pre-warm using RMAN physical validation."""
    dsn = oracledb.makedsn(host, port, service_name='ORCL')
    conn = oracledb.connect(user=user, password=password, dsn=dsn)
    cur = conn.cursor()

    cur.execute("""
        BEGIN
            rdsadmin.rdsadmin_rman_util.validate_current_dbfile(
                p_validation_type => 'PHYSICAL',
                p_parallel        => 4
            );
        END;
    """)

    cur.close()
    conn.close()
    return "RMAN physical validation complete"


def prewarm_sqlserver(host, port, user, password):
    """Pre-warm using DBCC CHECKDB for each database."""
    conn_str = (
        f"DRIVER={{ODBC Driver 17 for SQL Server}};"
        f"SERVER={host},{port};"
        f"UID={user};PWD={password}"
    )
    conn = pyodbc.connect(conn_str, autocommit=True)
    cur = conn.cursor()

    # Get all user databases
    cur.execute("""
        SELECT name FROM sys.databases
        WHERE database_id > 4
          AND state_desc = 'ONLINE'
          AND name NOT IN ('rdsadmin')
    """)
    databases = [row[0] for row in cur.fetchall()]

    for dbname in databases:
        try:
            logger.info(f"Running DBCC CHECKDB on {dbname}")
            cur.execute(
                f"DBCC CHECKDB ([{dbname}]) "
                f"WITH PHYSICAL_ONLY, NO_INFOMSGS"
            )
        except Exception as e:
            logger.error(
                f"DBCC CHECKDB failed for {dbname}: {e}"
            )

    cur.close()
    conn.close()
    return f"DBCC CHECKDB completed for {len(databases)} databases"


def prewarm_db2(host, port, user, password):
    """Pre-warm using INSPECT CHECK DATABASE."""
    conn_str = (
        f"DATABASE=BLUDB;HOSTNAME={host};PORT={port};"
        f"PROTOCOL=TCPIP;UID={user};PWD={password}"
    )
    conn = ibm_db.connect(conn_str, '', '')

    stmt = ibm_db.exec_immediate(
        conn,
        "CALL SYSPROC.ADMIN_CMD("
        "'INSPECT CHECK DATABASE RESULTS KEEP')"
    )

    ibm_db.close(conn)
    return "INSPECT CHECK DATABASE complete"

Lambda Configuration Notes:

  • Timeout: Set to the maximum (15 minutes) or use Lambda's streaming response. For very large databases where pre-warming exceeds 15 minutes, consider using AWS Fargate tasks instead of Lambda.
  • Memory: 512 MB–1 GB is sufficient; the Lambda is I/O-bound, not compute-bound.
  • VPC: Must be deployed in the same VPC as the RDS instances, with VPC endpoints for Secrets Manager and RDS.
  • Lambda Layers: Include the database driver libraries (psycopg2, oracledb, pyodbc, ibm_db) as Lambda layers. Alternatively, package them with the function code.
  • Concurrency: Set reserved concurrency to 200–500 to avoid exhausting the default 1,000 concurrent execution limit for other functions in the account.

Scaling to Hundreds of Databases

For 900 databases, use the Step Functions Map state with MaxConcurrency to process databases in controlled batches:

{
  "Type": "Map",
  "ItemsPath": "$.restored_databases",
  "MaxConcurrency": 200,
  "Iterator": {
    "StartAt": "HydrateOneDatabase",
    "States": {
      "HydrateOneDatabase": {
        "Type": "Task",
        "Resource": "arn:aws:states:::states:startExecution.sync:2",
        "Parameters": {
          "StateMachineArn": "arn:aws:states:...:hydration-per-db",
          "Input.$": "$"
        },
        "End": true
      }
    }
  }
}

This processes up to 200 databases concurrently. Each database goes through the full hydration workflow (credential retrieval → pre-warming → monitoring → IOPS scale-back → reporting) independently.

Why 200 and not 900? Several practical limits:

  • Lambda default concurrency is 1,000 per region. Running 900 hydration Lambdas simultaneously leaves only 100 for other functions
  • Secrets Manager GetSecretValue supports 10,000 TPS. This is not a bottleneck, but batching is still good practice
  • RDS API ModifyDBInstance has rate limits. 200 concurrent modifications is a safe operating point
  • Database engines have their own I/O limits. Running pre-warming on too many instances sharing the same underlying EBS infrastructure can cause contention

Downtime Considerations: The Scale-Up/Scale-Down Trade-off

If you choose to temporarily scale up the instance class for faster hydration (in addition to increasing IOPS), be aware of the downtime implications:

What Causes Downtime and What Doesn't

ModificationDowntime?
Increase/decrease IOPS on gp3 or io2No downtime. Online elastic volume operation. Volume enters optimizing state but remains available.
Increase/decrease throughput on gp3No downtime. Same as IOPS, elastic volume operation.
Change instance class (scale up or down)Yes, downtime. 5–15 minutes for Single-AZ; 1–2 minutes for Multi-AZ (failover-based).
Increase storage sizeNo downtime. But you cannot reduce storage size on RDS, the increase is permanent.
Change storage type (e.g., gp3 to io2)No downtime in most cases, but can be I/O-intensive.

The Zero-Downtime Path

If the customer restores to their production instance class but temporarily provisions maximum IOPS/throughput on gp3 or io2, runs pre-warming, then scales IOPS/throughput back down, there is zero downtime in the entire process. The trade-off is that pre-warming takes longer because the instance class EBS bandwidth is the bottleneck rather than the storage throughput.

For most instance classes at db.r6i.4xlarge and above (up to 10,000 Mbps = 1,250 MB/s), this is still fast enough for practical use.

The Fastest Path (With Downtime)

For maximum speed during a critical outage:

PhaseDurationDowntime?
1. Restore snapshot to larger instance + higher IOPS5–15 minNo, new instance being created
2. Run pre-warming commandDepends on DB size (see estimation table)No, database is online
3. Scale down IOPS/throughput to production levelsMinutes (elastic volume operation)No
4. Scale down instance class to production size5–15 min (Single-AZ) or 1–2 min (Multi-AZ)Yes

Important: There is a limit of four storage modifications within any 24-hour period on RDS. If you increase IOPS immediately after restore, you cannot modify storage again for 6 hours. Plan accordingly, ensure you set the right IOPS/throughput on the first modification.

RTO Benefit Summary

ScenarioWithout OptimizationWith Optimization (Zero-Downtime Path)With Optimization (Fastest Path)
500 GB on gp3 baselineHours (unpredictable)~30 min, zero downtime~10 min + 10 min downtime
1 TB on gp3 baseline3+ hours (unpredictable)~50 min, zero downtime~15 min + 10 min downtime
5 TB on gp3 baseline15+ hours (unpredictable)~4 hours, zero downtime~40 min + 10 min downtime

Looking Ahead

The techniques in this post provide practical, production-ready solutions for accelerating RDS snapshot hydration today. As AWS continues to evolve the RDS and EBS services, there are several areas where native enhancements could further simplify this workflow:

  • Native hydration acceleration in RDS. EBS Provisioned Rate for Volume Initialization (PRVI) is already available for EC2 workloads. Bringing similar capabilities natively into the RDS restore workflow would eliminate the need for engine-level pre-warming entirely.
  • Hydration completion events. An EventBridge event indicating that all EBS volumes backing an RDS instance are fully initialized would enable event-driven automation without CloudWatch polling.
  • Console visibility. A hydration progress indicator in the RDS console, similar to how storage optimization progress is displayed today, would improve operational visibility.
  • Broader Secrets Manager integration on restore. Extending --manage-master-user-password support on snapshot restore to all engines (currently limited to Oracle) would simplify credential management in automated restore workflows.

If any of these capabilities would benefit your workloads, I recommend filing a feature request through your AWS Support contact or Technical Account Manager.

Conclusion

The gap between "RDS instance available" and "RDS instance truly ready for production load" is a real operational blind spot that directly impacts RTO. While EBS features like Fast Snapshot Restore and Provisioned Rate for Volume Initialization solve this problem elegantly for self-managed EC2 databases, RDS customers today must work within the managed service abstraction.

The approach outlined in this post (combining engine-agnostic pre-warming commands, temporary IOPS scaling, CloudWatch-based hydration detection, secure credential management via the native RDS–Secrets Manager integration, and Step Functions orchestration, provides a practical, automatable, and reusable solution that works across PostgreSQL, Oracle, SQL Server, and Db2 at enterprise scale.

By implementing this pattern, you transform an unpredictable, hours-long hydration process into a predictable, observable workflow that can be measured, reported, and optimized, giving your teams confidence that when a database is declared ready, it truly is.

Related Resources

No comments

Relevant content