AWS re:Postを使用することにより、以下に同意したことになります AWS re:Post 利用規約
AWS re:Postre:Post

DMS IS NOT REPLICATING TO S3 THE NEW COLUMNS ADDED TO THE SOURCE DDL

0

I have created a replication task with Postgres as source and S3 as destination and I have noticed that it cannot replicate columns created after the start of the task. After the changes in the source table have been added, I create a task with exactly the same characteristics and in this way there is no problem to do the replication with the new columns Is there a way to capture the new columns without restarting the task? NOTE: The parameter that seems to work with this stays like this "HandleSourceTableAltered": true

**********************UPDATE: It is not enough to restart the task, apparently the only way to bring the new fields is by creating a new task only with the full load and with the CSV format, with Parquet the new columns are not migrated, if I activate the CDC configuration, neither I get the new columns

トピック
ストレージ移行と近代化データベース開発者用ツール
タグ
Amazon Simple Storage Service (S3)AWS Database Migration ServicePostgreSQL開発者用ツールデータの抽出、変換、ロード
言語
English
rePost-User-8799758
質問済み 1年前1418ビュー
2回答
1

When using AWS DMS, keep in mind that there are some limitations when it comes to capturing schema changes. When you add new columns to the source database, AWS DMS may not replicate these new columns automatically. As you have observed, restarting the replication task is usually required to resolve this issue. This is due to the replication task's need to reload the source tables' metadata in order to understand the new schema.

about the parameter yes that parameter can capture DDL events

https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TaskSettings.DDLHandling.html

https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TaskSettings.html#CHAP_Tasks.CustomizingTasks.TaskSettings.Example

profile picture
エキスパート
Sedat Salman
回答済み 1年前
  • rePost-User-8799758
    1年前

    Thanks for the answer, it is very helpful So what is the exact function of the HandleSourceTableAltered parameter? I have it set to True but it can't capture the changes in the DDL

  • rePost-User-8799758
    1年前

    I have noticed that restarting the task is not enough, at the moment the only solution is to create a new task with the same endpoints

  • AQM
    8ヶ月前

    I have an RDS MySQL source and a Redshift target. I added a column to a source table, but the task failed to capture it, and even after restarting the task many times, it is in Running with Errors state.

    My task settings have this:

     "ChangeProcessingDdlHandlingPolicy": {
        "HandleSourceTableDropped": true,
        "HandleSourceTableTruncated": true,
        "HandleSourceTableAltered": true
    },

    But still DMS is not capturing changes and task is in an error state.

    What's the solution to this? Surely, re-creating the task from start is not a viable option.

0

We had the exact same issue. After configuring

"CaptureDdls": true,

on DMS Postgres endpoint, new columns get captured instantaneously. Hope this works for you as well.

Matthias
回答済み 8ヶ月前

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン

関連するコンテンツ