DMS Migration Task is Not Creating Cdc-Files when source endpoint is S3

Question

In the past, a DMS migration tasks has sucessfully created the transaction files in the cdc-files folder when the target endpoint is configured as an S3 bucket.

After a redeploy and today, the tasks have stopped creating the cdc transaction files, however, the Updates and Updates Applied are correctly showing updates coming through cdc. Nothing has changed as far as I can see.

In the cloudwatch logs there is nothing indicating that files are being deposited.

![Updates Applied](/media/postImages/original/IMK8R92EUoTe-o6LLtu7izUQ)

Target Bucket Actual:
![No Files](/media/postImages/original/IMEbKCTAeSTz-SkEdtafCKGg)

Target Bucket Expected:
![Expected](/media/postImages/original/IMfoJMj0fdSa-Fb2jK71DxCg)

Edit:

For some reason unknown, the cdc-files are now being deposited to the full load directory. No changes can be found but looking into target endpoint settings at the moment. Any guidance AWS?

![Here is where](/media/postImages/original/IM4_HF4WCsTai0nuHcVz-mTQ)

Accepted Answer

Hello,

By default when AWS DMS runs a CDC task with S3 as a target, it stores all the row changes logged in your source database in one or more files for each table. Each set of files containing changes for the same table reside in a single target directory associated with that table. AWS DMS creates as many target directories as database tables migrated to the Amazon S3 target endpoint.

- This behavior is being noticed in the screenshot where both the full load and CDC .csv files reside in the same directory inside the S3 bucket.

Now, to capture source database changes in a manner that also captures the transaction order, you can specify S3 endpoint settings that direct AWS DMS to store the row changes for all database tables in one or more .csv files. These CDC files reside together in a single transaction directory that is specified on the S3 target endpoint settings. These CDC transaction .csv files follow the naming convention given below:

CDC_TXN-timestamp.csv

- This can be noticed in the "Target Bucket Expected" screenshot, as the .csv files reside in a different directory inside the S3 bucket and also follow the naming convention above.

Please refer to the following documentation for more information:
[+] https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.S3.html#CHAP_Target.S3.EndpointSettings.CdcPath

To tell AWS DMS to store all row changes in transaction order ie. to store all CDC changes in a separate directory please make sure that the following are settings are correctly configured for the S3 endpoint:

1. Set the PreserveTransactions S3 setting on the target to true.
2. Set the CdcPath S3 setting on the target to a relative folder path where you want AWS DMS to store the .csv transaction files.

Note: AWS DMS creates this path either under the default S3 target bucket and working directory or under the bucket and bucket folder that you specify using the BucketName and BucketFolder S3 settings on the target.

Please feel free to open a support case with AWS using the following link in case account-specific help is required:
[+] https://console.aws.amazon.com/support/home#/case/create

DMS Migration Task is Not Creating Cdc-Files when source endpoint is S3

Relevant content