Why doesn't my Amazon DocumentDB storage shrink after I delete documents?

Short description

Amazon DocumentDB does not release allocated storage space when individual documents are deleted from a collection. Instead, the freed space is marked as reusable for future document insertions within the same collection. This behavior results in a "high water mark" effect per entity, where the total allocated storage on the cluster volume (measured by the VolumeBytesUsed CloudWatch metric) remains at its historical peak, even if the actual live data is significantly smaller. Storage is only reclaimed when you drop an entire entity such as collection / index / database or perform a logical dump and restore.

Resolution

Understand how Amazon DocumentDB manages storage after document deletions

Amazon DocumentDB uses a distributed storage architecture that automatically grows as data is added. Starting with Amazon DocumentDB 4.0, when data is removed, the overall allocated space decreases. However, this storage reclamation behavior applies only when you drop an entire collection, index, or database. It does not apply to the deletion of individual documents.

When you delete individual documents (including deletions performed by a TTL index), the following occurs:

• The storage allocated for the deleted documents is not returned to the cluster volume.
• The freed space is internally marked as reusable by Amazon DocumentDB for the same collection.
• Future document insertions reuse this space before additional storage is allocated.
• The VolumeBytesUsed metric (which also determines snapshot and backup size) remains at the high water mark.

This means that even if you perform a bulk delete operation that removes a large percentage of your data, the VolumeBytesUsed metric does not decrease. The space is retained for reuse by future writes.

Understand Multi-Version Concurrency Control (MVCC) and dead document versions

Amazon DocumentDB uses Multi-Version Concurrency Control (MVCC) to manage concurrent read and write operations. When a document is updated or deleted, Amazon DocumentDB does not overwrite or physically remove the original document. Instead, it creates a new version of the document (for updates) or marks the existing version as a dead version (for deletes). The previous versions are retained in both the collection storage and all associated indexes.

This MVCC architecture has the following impact on storage:

• Every update operation creates a new document version. The old version becomes a dead version that still occupies storage.
• Every delete operation marks the document as a dead version rather than physically removing it from storage.
• Index entries associated with dead document versions also remain in the index structure as dead index entries.
• Over time, these dead versions accumulate and consume storage space that is not reflected in the logical data size reported by commands like db.stats().

Amazon DocumentDB has an internal garbage collection process that periodically sweeps and removes dead document versions. This removes dead versions to allow the storage pages to be reused, but the overall allocated storage does not shrink as a result.

To view the MVCC overhead on a collection, run the following command:

db.runCommand({collStats: "<collection_name>"})

Review the MVCCIdStats field in the output. The MVCCIdScale value indicates the ratio of MVCC overhead relative to the collection size. A higher value indicates more dead versions are present in the collection.

To determine how much unused storage exists in your collections, run the following command on each collection:

db.runCommand({collStats: "<collection_name>"})

Review the following fields in the output:

• size: The actual uncompressed size of the data in the collection.
• storageSize: The total amount of storage allocated for the collection, including unused space.
• unusedStorageSize: The amount of allocated storage that is not currently being used (bloat). This field reports unusedBytes and unusedPercent values.

A large difference between size and storageSize indicates significant storage bloat in the collection.

To check for index bloat, run the following command:

db.<collection_name>.aggregate([{$indexStats: {}}])

Review the following fields in the output for each index:

• size: The total size of the index.
• accesses.ops: The number of operations that have used this index. A value of 0 indicates an unused index.
• unusedStorageSize: The amount of unused space within the index. This field reports unusedSizeBytes and unusedSizePercent values.

An unusedSizePercent value above 50% indicates significant index bloat that is contributing to increased storage consumption.

If your cluster runs Amazon DocumentDB 5.0 or later, use the reIndex command to rebuild indexes and reclaim unused index storage without performing a full dump and restore.

To rebuild a specific index in a collection, run the following command:

db.runCommand({ reIndex: "<collection-name>", index: "<index-name>"})

The reIndex command is an online operation and does not require downtime. However, it consumes cluster resources (CPU and I/O). Schedule reIndex operations during periods of low traffic to minimize any performance impact. Amazon DocumentDB supports reindex of a single index in the background, allowing for multiple workers. The old index is usable by queries when the reIndex process is running.

If your cluster runs Amazon DocumentDB 4.0, the reIndex command is not available. Instead, drop the bloated index and recreate it with the same definition to reclaim unused index storage. During the time between dropping and recreating the index, queries that depend on that index experience performance degradation because they fall back to collection scans. Perform this operation during a maintenance window or a period of low traffic.

Related information

Troubleshooting performance and resource utilization

Database index bloat in Amazon DocumentDB

Garbage collection in Amazon DocumentDB

Best practices for Amazon DocumentDB

Amazon DocumentDB quotas and limits