Zero-ETL Integration - CDC creates duplicate records in S3, causing data inconsistency in Glue Data Catalog

Question

I am currently setting up a Zero-ETL integration pipeline from DynamoDB to S3 using AWS Glue in the ap-northeast-1 region and I am experiencing a data duplication issue during the CDC (Change Data Capture) phase.

Setup:
- Source: DynamoDB table ("course")
- Target: S3 bucket via AWS Glue Zero-ETL Integration
- Data Catalog: AWS Glue with Lake Formation enabled
- Query engine: Amazon Athena

Issue:
After the initial SEED/full load completes successfully, the CDC process appends new Parquet files to S3 instead of overwriting the existing ones when records are updated in DynamoDB. When I re-run the Glue Crawler, it picks up all Parquet files (both old and new), resulting in duplicate records in the Glue Data Catalog table — one with the old values and one with the updated values.

Questions:
1. Is this the expected behavior for Zero-ETL CDC with DynamoDB as the source?
2. What is the AWS-recommended approach to handle deduplication in this scenario — should I use Glue ETL jobs, Apache Iceberg tables, or another service?
3. Does AWS Glue Zero-ETL support Iceberg format natively for DynamoDB as source so that UPSERT/MERGE operations are handled automatically?
4. Are there any best practices or reference architectures for this specific use case?

Thank you for your support.

Florian Turnwald · Accepted Answer

As to my understanding, the data duplication issue is caused by using an AWS Glue Crawler on the target S3 bucket.

Here is what I think, and is happening and how to fix it …

* **Root Cause:** AWS Glue Zero-ETL integrations use the **Apache Iceberg** table format natively to handle UPSERT and MERGE operations automatically. Iceberg uses a metadata layer to track the latest state of your records, even though old Parquet files remain in S3. When you run a standard AWS Glue Crawler directly on the S3 bucket, it bypasses the Iceberg metadata and reads all raw Parquet files (both the initial load and all historical CDC updates). This is why you see old and new values as duplicate records in Athena.
 * **Solution:** You do not need (and should not use) a Glue Crawler for this pipeline. The Zero-ETL integration automatically creates and maintains the destination table in the AWS Glue Data Catalog as an Apache Iceberg table. To see the correct, deduplicated data, simply query this automatically registered Iceberg table directly in Amazon Athena.
 * **Native Support:** Yes, the integration natively supports Iceberg. You do not need to build manual AWS Glue ETL jobs for deduplication.

**Official Reference:**
You can verify how Athena relies on Iceberg's metadata to resolve the latest state of records (and why crawling raw files breaks this) in the official AWS documentation: https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html

re:Post Agent · Answer

The behavior you're experiencing is expected for Zero-ETL integrations with DynamoDB as a source. The CDC process is designed to append change records to S3 rather than overwrite existing files, which can result in multiple versions of the same record appearing in your Glue Data Catalog when queried through Athena.

To address the deduplication challenge, here are the recommended approaches:

**Deduplication Strategy:**
The most common approach is to use AWS Glue ETL jobs to handle deduplication. You can create a Glue job that reads the data from S3, identifies the most recent version of each record (typically using a timestamp or version field from the CDC data), and writes the deduplicated results to a separate location or table. This gives you control over the deduplication logic and allows you to maintain a clean, queryable dataset.

**Data Quality Checks:**
AWS Glue Data Quality includes features to help identify duplicate records. You can use the FileUniqueness rule to detect duplicate files in your S3 folders, though this focuses on file-level rather than record-level duplicates. For record-level deduplication, you'll need to implement custom logic in your ETL jobs.

**Refresh Interval Considerations:**
Be aware that for DynamoDB sources with refresh intervals of 1440 minutes (24 hours) or greater, the integration uses sequential daily batch processing instead of a single export operation. This is due to DynamoDB's export window limitation. The integration will wait for the full refresh interval, then perform multiple sequential exports covering up to 24-hour windows each, with CDC jobs processing each batch sequentially.

**Best Practices:**
- Implement a Glue ETL job that runs after CDC updates to deduplicate records based on primary keys and timestamps
- Consider partitioning your target S3 data by date or other relevant dimensions to improve query performance
- Use the AWS Glue Data Catalog's versioning capabilities to track schema changes
- Set up appropriate refresh intervals based on your data freshness requirements and the volume of changes

Regarding Apache Iceberg support: While AWS Glue does support Apache Iceberg table format for certain use cases, you should verify the current capabilities for Zero-ETL integrations with DynamoDB specifically, as native UPSERT/MERGE handling may require additional configuration or ETL processing.

For your specific architecture, the recommended pattern is: DynamoDB → Zero-ETL Integration → S3 (raw CDC data) → Glue ETL Job (deduplication) → S3 (curated data) → Glue Data Catalog → Athena queries.  
**Sources**  
[Configuring an integration - AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/zero-etl-configuring-integration.html)  
[FileUniqueness - AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/dqdl-rule-types-FileUniqueness.html)

Zero-ETL Integration - CDC creates duplicate records in S3, causing data inconsistency in Glue Data Catalog

Add your answer

Relevant content