Does Migration Tasks Targeting S3 as Source Ignore CDC_TXN Files Previously Processed on a Reload
When an S3 is the Source of a full-load and Ongoing replication task, it seems that a "Resume" always uses a pointer to pick up where it last left off. Does a "Restart" also ignore previously processed CDC files?
- Newest
- Most votes
- Most comments
Hello,
When you use a full load and CDC task, AWS DMS migrates table data, and then applies data changes that occur on the source. If you restart the task, this loads all the tables again, and starts capturing source changes from the restart time i.e. the previously processed CDC_TXN files (Transactions files for S3 as target) will be ignored while capturing and applying the new transaction files after the restart.
Reloading tables during a task: [+] https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.ReloadTables.html
Further, AWS DMS also uses the Table preparation mode that you chose when you created the task. Table preparation modes include Drop table on target, Truncate, and Do nothing. If your task is configured with the Do nothing preparation mode, then consider manually emptying the target tables before restarting the task to avoid duplicate errors.
Full-Load Task settings: [+] https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TaskSettings.FullLoad.html
Please feel free to open a support case with AWS using the following link in case account-specific help is required: [+] https://console.aws.amazon.com/support/home#/case/create
Relevant content
- Accepted Answerasked a year ago
AWS OFFICIALUpdated a year ago- Why is no data migrated from my Amazon S3 source endpoint even though my AWS DMS task is successful?AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 6 months ago
It sounds like a full-load will ignore already processed cdc files. I think the problem was that there was some load testing going on and a full load was applied to the task and there were transactions from the prior full load that still needed to be processed after restart. It appeared as if reload was loading all cdc files in the directory, however, I will circle back with QA.