Compare specific fields for 3 milion documents in two different DocumentDBs clusters

In addition to the suggestions above, it is worth clarifying that DocumentDB (unlike DynamoDB) does not feature native "segmented scans". However, for 3 million documents, complex parallelization is likely overengineering.

Since you only need to compare three fields, using projection (find({}, {"A":1, "B":1, "C":1})) reduces the total payload to a few hundred megabytes at most.

The most straightforward and performant approach:

• Open a standard MongoDB cursor on both clusters, sorted by _id.
• Set a high batch size (e.g., cursor.batch_size(10000)) to minimize network round-trips.
• Iterate through both cursors synchronously in a single script.

Because field-level encryption was applied, your actual performance bottleneck will not be the database read speed, but the CPU time needed by your script to decrypt Cluster B's data before comparison. Running this comparison script on a compute-optimized EC2 instance is highly recommended.

Reference on DocumentDB query best practices (specifically around using Projections to minimize data transfer): https://docs.aws.amazon.com/documentdb/latest/developerguide/best_practices.html#best_practices-query_optimization

For comparing specific fields across millions of documents between two DocumentDB clusters, especially after a migration with field-level encryption, here are the most performant approaches:

Recommended Approach: Parallel Scanning with Batch Processing

The most efficient method involves running parallel scans against both clusters and comparing documents in batches. This approach is linear in time, highly parallelizable, and uses minimal memory. Key implementation considerations:

1. Use segmented scans to parallelize the comparison across multiple threads or processes. This allows you to divide the workload and significantly reduce total duration.

2. Implement batch processing where you fetch documents in batches (e.g., 1000-5000 documents at a time) rather than one-by-one. This reduces network round trips and improves throughput.

3. Compare only the specific fields you need (A, B, and C) rather than entire documents. Project only these fields in your queries to reduce data transfer.

4. Leverage the _id for ordering since your _id values are preserved during migration. Sort by _id in both scans to enable efficient sequential comparison.

Performance Optimization Tips:

• Run during quiet periods when write operations are minimal to ensure consistency and avoid resource contention
• Monitor and adjust resource usage by tuning batch sizes and thread counts based on your cluster's capacity
• Consider using indexes on the _id field (which exists by default) to speed up lookups
• Implement checkpointing so you can resume if the comparison process is interrupted

Expected Duration:

For 1-3 million documents, with proper parallelization and batch processing, you should be able to complete the comparison in minutes to under an hour, depending on document size, network latency between accounts, and the degree of parallelization. The comparison will likely take longer than your migration script since you're performing field-level validation rather than bulk copying.

Implementation Pattern:

Your comparison logic should:

1. Initialize parallel scan streams for both clusters
2. Fetch batches of documents sorted by _id
3. Compare the specified fields (A, B, C) for matching _id values
4. Log any discrepancies for review
5. Track progress and maintain metrics on comparison rate

Since you mentioned field-level encryption was applied during migration, ensure your comparison accounts for this - you may need to decrypt fields in cluster B before comparing, or compare encrypted values if that's appropriate for your use case.
Sources
Amazon DocumentDB migration runbook - Amazon DocumentDB
Efficiently compare items across two Amazon DynamoDB tables | AWS Database Blog