DMS instance version 3.6.1 is not replicating source table headers with some database sources

Question

I migrate tables from RDS MariaDB to Redshift using DMS 3.5.3 instance version and I add AR_H_TIMESTAMP source table header using transformation rules, it works fine with this DMS version, however when I updated the instance version from 3.5.3 to 3.6.1, it didn't work and actually it keeps the column value of this variable as null in the destination table, I tried to do the same with the 3.6.1 migration instance version, but with RDS PostgreSQL as source and it worked fine.

TL;DR source table headers works with 3.5.3 DMS instance version with RDS MariaDB as source, but doesn't work with 3.6.1 DMS instance version however it works with RDS PostgreSQL with the same version.

TL;DR
source table headers works with 3.5.3 DMS instance version with RDS MariaDB as source, but doesn't work with 3.6.1 DMS instance version however it works with RDS PostgreSQL with the same version.

re:Post Agent · Answer

Based on your description, it appears you're experiencing an issue with source table header replication when using AWS DMS version 3.6.1 specifically with MariaDB as a source, while the same configuration works with PostgreSQL sources.

This behavior could be related to how different database sources handle metadata during the replication process. By default, headers for source tables aren't replicated to the target, and you need to use transformation rules with expressions that include the table column header to indicate which headers to replicate.

When using MariaDB as a source, there are some specific limitations that might affect header replication. For example, AWS DMS does not support views migration for MariaDB, and there are limitations around temporal data tables. Additionally, the way MariaDB handles binary logging (which is essential for CDC) differs from PostgreSQL's logical replication mechanism.

For the specific header `AR_H_TIMESTAMP`, this should contain a timestamp indicating the time of the change during ongoing replication, or the current time data arrives at the target during full load. The data type for this header is DATETIME with a scale of 7.

To troubleshoot this issue:

1. Verify that your transformation rule is correctly configured to add the `AR_H_TIMESTAMP` column
2. Check if there are any errors or warnings in the CloudWatch logs (search for "]E:" for errors and "]W:" for warnings)
3. Ensure that binary logging is properly enabled on your MariaDB source with `binlog_format` set to `ROW`
4. Confirm that the AWS DMS user has all the required permissions on the MariaDB source

If the issue persists, you might need to consider either reverting to DMS version 3.5.3 for your MariaDB migrations or opening a support case with AWS to investigate if there's a specific compatibility issue between DMS 3.6.1 and MariaDB source table headers.  
**Sources**  
[Using transformation rule expressions to define column content - AWS Database Migration Service](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TableMapping.SelectionTransformation.Expressions.html)  
[Using a MySQL-compatible database as a source for AWS DMS - AWS Database Migration Service](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.MySQL.html)  
[Postgres to Redshift Full Load and CDC does not capture updates | AWS re:Post](https://repost.aws/questions/QU0nSlj53_ScmzEpCsu0EE0w/postgres-to-redshift-full-load-and-cdc-does-not-capture-updates)  
[DMS Migration Task(only CDC) for Ongoing Replication Type, Migration progress struck at 0% | AWS re:Post](https://repost.aws/questions/QUp-mHF70-QF6xXPe-_Iepng/dms-migration-task-only-cdc-for-ongoing-replication-type-migration-progress-struck-at-0)

DMS instance version 3.6.1 is not replicating source table headers with some database sources

Add your answer

Relevant content