How do I troubleshoot garbage collection issues on my Amazon DocumentDB cluster?

Resolution

Amazon DocumentDB uses Multi-Version Concurrency Control (MVCC) to manage concurrent transactions. With MVCC, when documents are updated or deleted, their previous versions remain in collections and indexes as "dead" versions. The garbage collection (GC) process automatically reclaims space from these dead versions for future operations. Each write operation consumes a unique MVCC ID from a finite counter. Over time, these old versions and their MVCC IDs accumulate, requiring cleanup to prevent performance degradation .

Common Causes of Garbage Collection Issues

1. Long-Running Queries

Active queries or cursors block the garbage collector from purging old document versions. The GC must wait for these operations to finish to avoid disrupting consistency. Thus long-running queries leads to high CPU and can create memory pressure, forcing the garbage collector to work harder and delaying the cleanup of outdated documents. This increased workload can lead to further performance degradation and exacerbate garbage collection delays, creating a cycle of inefficiency. By avoiding long-running transactions and closing cursors promptly, you enable the garbage collector to make consistent progress and maintain a healthy database environment.

Use the following command to find long-running queries:

 // To find a query running for more than 30 mins
db.adminCommand({
    aggregate: 1,
    pipeline: [
        {$currentOp: {}},
        {$match: 
            {$or: 
                [{secs_running: {$gt: 1800}},
                 {WaitState: {$exists: true}}]}}],
    cursor: {}
});

Use the following command to find open cursors running for long:

// To find cursor which is running more than 30 mins
db.adminCommand({
    "currentOp": true,
    "active": true,
    "$all": true
}).inprog.filter(function(op) {
    return op.desc == "Cursor" && 
           op.secs_running > 1800 && 
           op.active == true;
}).sort((a, b) => b.microsecs_running - a.microsecs_running)

Use the following command to find idle sessions with open transactions:

db.adminCommand({
  aggregate: 1,
  pipeline: [
    { $currentOp: { allUsers: true, idleSessions: true } },
    { $match: { active: false, transaction: { $exists: true } } }
  ],
  cursor: {}
})

Kill Long-Running Operations

Firstly, list all running operations on an Amazon DocumentDB instance, by using the following command:

db.adminCommand({currentOp: 1, $all: 1});

It returns a complete list of all user queries and internal system tasks currently operating on the instance. The output includes details such as the operation description, whether it is active, the operation ID, the namespace, and the time running in seconds and microseconds.

In the output, the desc: GARBAGE_COLLECTION entries indicate that garbage collection is actively running on specific collections. The garbageCollection object within these entries specifies the database and collection names involved in the garbage collection process. For example:

{
"desc": "GARBAGE_COLLECTION",
"garbageCollection": {
"databaseName": "testdb",
"collectionName": "testCollectionA"
},
"secs_running": 3,
"microsecs_running": NumberLong(3123456)},
{
"desc": "GARBAGE_COLLECTION",
"garbageCollection": {
"databaseName": "testdb",
"collectionName": "testCollectionB"
},
"secs_running": 4,
"microsecs_running": NumberLong(4123456)
}

Here, testCollectionA and testCollectionB in the collectionName fields are the collections in the testdb database that are currently undergoing garbage collection. This direct link between the collection name in the output and the collection in a long-running query helps identify which specific collections are impacting performance.

Once identified, terminate the long-running transactions with caution. It is essential to ensure that the operations being terminated are non-critical and do not impact essential processes. Always investigate the context and potential impact before termination to avoid unintended consequences.

// Kill a specific operation by opId
db.adminCommand({ killOp: 1, op: <opId> });

2. High Write/Update/Delete Throughput

When your Amazon DocumentDB cluster experiences rapid document modifications including updates, and deletes, it creates multiple versions of documents due to Multi-Version Concurrency Control (MVCC). If the rate at which these document changes occur surpasses the garbage collector's ability to clean up old versions, storage consumption will begin to grow. This growth happens because the GC process can only clean up old document versions at a certain pace, which depends on factors such as your cluster size, available CPU resources, and the complexity of the documents being processed. When incoming changes outpace this cleanup capability, you'll see increased storage usage that can eventually lead to storage fragmentation if not addressed.

Here are the recommended approaches to mitigate the issue:

a. Throttle or Batch Write Operations

• Spread bulk operations over time rather than executing large batch deletes/updates in a single burst
• Implement application-level rate limiting for write-heavy workloads
• Use smaller batch sizes (e.g., 500–1,000 documents per batch) with pauses between batches

b. Scale Up the Instance Class

• Upgrade to a larger instance type. Larger instances provide more CPU and memory for background GC processes to keep up with write throughput.
• Additionally, consider leveraging NVMe storage capabilities that offer significantly improved read requirements compared to traditional network storage.

3. Replica Lag Read replicas that fall behind can hold references to old document versions, delaying GC on the primary.

Common Causes of Replica Lag & Fixes

• Replica Instance Under-Provisioned: Replica instances showing consistently high CPU utilization (>80%) with steadily growing lag, scale up the replica instance class to provide additional processing power.
• Long-Running Queries on Replicas:To address long-running queries on replicas where expensive aggregations or full collection scans hold old snapshots. Implement maxTimeMS limits on all replica-targeted queries and terminate operations that exceed reasonable execution durations.

Monitoring Garbage collection

Effective monitoring of garbage collection (GC) is crucial for maintaining optimal cluster performance. Key metrics help identify when GC is struggling to keep pace with your workload.

AvailableMVCCIds

• Tracks how many write operations your Amazon DocumentDB cluster can still process before reaching its maximum limit of 1.8 billion.
• Each write operation consumes one MVCC ID, and when this counter hits zero, your cluster automatically enters read-only mode until garbage collection can reclaim and recycle these IDs.

The wrap-around issue occurs when AvailableMVCCIds reaches zero because the cluster has exhausted all available MVCC IDs. At this point, new write operations are blocked, and your database becomes read-only. This can severely impact application functionality, especially in write-intensive environments.

LongestActiveGCRuntime

• Shows the duration (in seconds) of the longest currently running garbage collection process.
• To effectively use this metric, compare current values against historical data from gcRuntimeStats. This comparison helps identify abnormal garbage collection behavior, such as unusually extended runtimes that might occur during bulk deletions or other intensive operations.

Monitoring LongestActiveGCRuntime allows you to detect when garbage collection processes are taking longer than expected, providing early warning signs of potential performance bottlenecks or systemic issues in your Amazon DocumentDB cluster.

MVCCIdScale

• Accessible through the database collStats command, measures MVCC ID age on a normalized scale from 0 to 1 where 1 represents the maximum age before your Amazon DocumentDB cluster enters read-only state.
• It identify which collections contain the oldest MVCC IDs that contribute to overall cluster aging. For effective management, use MVCCIdScale in conjunction with the AvailableMVCCIds metric to get a complete picture of your cluster's MVCC health.

To prevent any single collection from becoming a bottleneck for garbage collection, maintain MVCCIdScale values below 0.3 for each collection, ensuring balanced aging across your database and preventing premature wrap-around issues.

Analyzing documentFragmentStats

The documentFragmentStats metric, available through the database collStats command, provides comprehensive insights into document fragments and dead data within your Amazon DocumentDB collections. Document fragments are the internal storage units used by the database engine, while dead fragments represent data that is no longer accessible but hasn't yet been reclaimed by garbage collection.

This metric delivers several key measurements:

• totalDocFragmentsCount shows the overall number of document fragments in a collection
• deadDocFragmentsCount counts fragments containing inaccessible data
• deadDocFragmentsPercent calculates what percentage of fragments are dead
• deadDocFragmentBytes estimates storage consumed by these unreclaimed fragments.

Additionally, it offers a per-segment breakdown for both storageSegmentBase and storageSegmentExtended, helping you identify where dead data is accumulating and assess the effectiveness of your garbage collection processes.

Related information

Garbage collection in Amazon DocumentDB - Amazon DocumentDB.

Long running queries - Amazon DocumentDB.

NVMe-backed instances - Amazon DocumentDB.

Troubleshooting performance and resource utilization.