Why am I seeing high IOPS or increased I/O cost on my Amazon DocumentDB cluster?

Short description

Amazon DocumentDB charges for I/O when data is read from or written to the storage volume. Each read operation counts as one I/O per 8 KB page, and each write operation counts as one I/O per 4 KB unit. Data served from the buffer cache does not incur I/O charges. High IOPS typically results from one or more of the following causes:

• Your working set exceeds available instance memory leading to high read IOPS
• Queries perform full collection scans due to missing indexes
• Garbage collection processes old document versions, generating read IOPS
• Excessive indexes amplify write I/O
• TTL deletions generate write and read I/O

Resolution

Determine whether your working set fits in memory

Amazon DocumentDB reserves one-third of instance RAM for engine services. Only two-thirds is available for the buffer cache. When your working set exceeds the available cache, reads spill to the storage volume and generate I/O charges.

1. In the CloudWatch console, review the BufferCacheHitRatio metric for each instance.
2. If this metric is consistently below 95%, your working set does not fit in memory.
3. Scale up your instance class to provide more RAM. Continue scaling until the metric no longer increases dramatically.
4. For workloads with large datasets that exceed memory, consider NVMe-backed instances. These instances use local NVMe SSD as a second-tier cache, holding up to five times more data than the buffer cache alone. This reduces storage I/O when the working set spills beyond RAM. NVMe-backed instances are particularly beneficial for read-heavy workloads with low BufferCacheHitRatio and high ReadIOPS, and for update-heavy workloads where garbage collection is limited by storage read latency.

Identify and resolve collection scans

When a query lacks an index, Amazon DocumentDB performs a full collection scan (COLLSCAN), reading every document from storage.

1. Enable the profiler by setting the profiler parameter to enabled and profiler_threshold_ms to 100 in your cluster parameter group. Then enable profiler log exports to CloudWatch Logs.
2. In CloudWatch Logs Insights, query the profiler logs for collection scans:

filter planSummary="COLLSCAN"

If the profiler output identifies queries performing collection scans, create an appropriate index on the filter fields used by those queries.

Reduce garbage collection I/O

Amazon DocumentDB uses multi-version concurrency control (MVCC). Each update creates a new document version, and garbage collection (GC) removes old versions automatically. When GC runs, it loads obsolete document and index versions into memory and deletes them. This generates read IOPS on the storage volume.

To identify whether GC is contributing to your read IOPS:

1. Monitor the LongestActiveGCRuntime CloudWatch metric. This shows the duration in seconds of the longest active garbage collection process. Correlate periods where LongestActiveGCRuntime is elevated with spikes in your VolumeReadIOPs metric. If they align, GC is driving the read I/O.
2. Identify which collections are contributing to GC load at the collection level:

db.collection.stats()

Review gcRuntimeStats (number of runs, average and max duration), documentFragmentStats.deadDocFragmentsPercent (investigate if above 10%), and MVCCIdStats.MVCCIdScale (maintain below 0.3). Collections with high dead fragment percentages or long GC runtimes are generating the most I/O.

3. If GC is falling behind, scale up your instance to give the garbage collector more CPU and memory resources.

Reduce I/O from TTL deletions

TTL deletions incur I/O per document deleted and create dead MVCC versions that require additional GC I/O to clean up. Monitor the TTLDeletedDocuments CloudWatch metric to understand the volume of TTL deletions happening on your cluster. Replace TTL indexes with rolling collections where possible. Segment documents into time-based collections and drop them when no longer needed. Dropping a collection does not incur per-document I/O.

Reduce write I/O from excessive indexes

Each index on a collection adds one extra write I/O per document insert, update, or delete. For example, if a collection has nine indexes, the database must perform ten writes (one for the document plus one per index) before acknowledging the operation.

1. Identify unused indexes:

db.collection.aggregate([{$indexStats: {}}]).pretty()

Drop indexes where accesses.ops equals zero. Regularly reviewing and removing unused indexes reduces unnecessary compute, storage, and I/O.

2. As per the best practices for working with indexes, keep only the indexes that are necessary to improve performance for your common queries. Only create indexes on fields commonly used as filters, and target fields where the typical value frequency is 1% or less of the total collection size.

Reduce storage and I/O using document compression

If your cluster runs Amazon DocumentDB 5.0 or later, turn on document compression at the collection level to reduce the physical size of stored documents, which lowers both storage costs and I/O. Compressed documents occupy fewer 8 KB pages on disk, so fewer read I/O operations are needed to retrieve them and fewer write I/O operations to persist them. Note that document compression does not address fragmentation caused by deletions. Use rolling collections for that.

Reduce I/O cost with I/O-Optimized storage

If your workload is I/O-intensive and I/O charges exceed 25% of your total cluster cost, switch to I/O-Optimized storage configuration to eliminate per-I/O charges:

aws docdb modify-db-cluster --db-cluster-identifier your-cluster --storage-type iopt1

There is no downtime when switching. You can switch to I/O-Optimized once every 30 days, and switch back to Standard at any time.

Related information

Garbage collection in Amazon DocumentDB

Best practices for Amazon DocumentDB

Amazon DocumentDB cluster storage configurations

NVMe-backed instances

Profiling Amazon DocumentDB operations