By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

DMS IS NOT REPLICATING TO S3 THE NEW COLUMNS ADDED TO THE SOURCE DDL

0

I have created a replication task with Postgres as source and S3 as destination and I have noticed that it cannot replicate columns created after the start of the task. After the changes in the source table have been added, I create a task with exactly the same characteristics and in this way there is no problem to do the replication with the new columns Is there a way to capture the new columns without restarting the task? NOTE: The parameter that seems to work with this stays like this "HandleSourceTableAltered": true

**********************UPDATE: It is not enough to restart the task, apparently the only way to bring the new fields is by creating a new task only with the full load and with the CSV format, with Parquet the new columns are not migrated, if I activate the CDC configuration, neither I get the new columns

Topics
StorageMigration & ModernizationDatabaseDeveloper Tools
Tags
Amazon Simple Storage ServiceAWS Database Migration ServicePostgreSQLDeveloper ToolsExtract Transform & Load Data
Language
English
rePost-User-8799758
asked a year ago1372 views
2 Answers
1

When using AWS DMS, keep in mind that there are some limitations when it comes to capturing schema changes. When you add new columns to the source database, AWS DMS may not replicate these new columns automatically. As you have observed, restarting the replication task is usually required to resolve this issue. This is due to the replication task's need to reload the source tables' metadata in order to understand the new schema.

about the parameter yes that parameter can capture DDL events

https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TaskSettings.DDLHandling.html

https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TaskSettings.html#CHAP_Tasks.CustomizingTasks.TaskSettings.Example

profile picture
EXPERT
Sedat Salman
answered a year ago
  • rePost-User-8799758
    a year ago

    Thanks for the answer, it is very helpful So what is the exact function of the HandleSourceTableAltered parameter? I have it set to True but it can't capture the changes in the DDL

  • rePost-User-8799758
    a year ago

    I have noticed that restarting the task is not enough, at the moment the only solution is to create a new task with the same endpoints

  • AQM
    8 months ago

    I have an RDS MySQL source and a Redshift target. I added a column to a source table, but the task failed to capture it, and even after restarting the task many times, it is in Running with Errors state.

    My task settings have this:

     "ChangeProcessingDdlHandlingPolicy": {
        "HandleSourceTableDropped": true,
        "HandleSourceTableTruncated": true,
        "HandleSourceTableAltered": true
    },

    But still DMS is not capturing changes and task is in an error state.

    What's the solution to this? Surely, re-creating the task from start is not a viable option.

0

We had the exact same issue. After configuring

"CaptureDdls": true,

on DMS Postgres endpoint, new columns get captured instantaneously. Hope this works for you as well.

Matthias
answered 7 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content