Issue with AWS DMS Full Load + CDC Task Not Reflecting Newly Created Tables in Target DB

0

Dear Team, I am writing to seek assistance regarding an issue I am encountering with an AWS DMS task configured for full load plus Change Data Capture (CDC). Currently, I am using AWS DMS to migrate data from a PostgreSQL aurora source database to a target PostgreSQL aurora database. The task is configured to perform both full load and CDC. However, I have observed that while existing data from all tables is successfully replicated to the target database, newly created tables in the source database are not being reflected in the target database. To summarize the issue:

  • Existing tables and their data from the source database are successfully replicated to the target database.
  • Newly created tables in the source database are not replicated to the target database.
1 Antwort
1
Akzeptierte Antwort

Ensure that your table mapping rules include the newly created tables. If your task is configured with specific table mappings, new tables won't be automatically included unless the mappings are updated.

Update your table mapping rules to include the new tables. You can use wildcard patterns to include all tables dynamically.

example

{
  "rules": [
    {
      "rule-type": "selection",
      "rule-id": "1",
      "rule-name": "1",
      "object-locator": {
        "schema-name": "%",
        "table-name": "%"
      },
      "rule-action": "include"
    }
  ]
}

if above condition is already in use. check if the the ddl statement being used is supported by DMS. also by default ddls are not reported by pg in wal logs To capture DDL events, AWS DMS creates various artifacts in the PostgreSQL database when a migration task starts.

check if you have below objects in source db under public schema

trigger awsdms_intercept_ddl;
function awsdms_intercept_ddl
table awsdms_ddl_audit

another reason for not capturing ddls as mentioned in limitations of psql as source is In most cases, AWS DMS supports change processing of CREATE, ALTER, and DROP DDL statements for tables. AWS DMS doesn't support this change processing if the tables are held in an inner function or procedure body block or in other nested constructs.

For example, the following change isn't captured.

CREATE OR REPLACE FUNCTION attu.create_distributors1() RETURNS void
LANGUAGE plpgsql
AS $$
BEGIN
create table attu.distributors1(did serial PRIMARY KEY,name
varchar(40) NOT NULL);
END;
$$;

Also If your source database is also a target for another third–party replication system, DDL changes might not migrate during CDC. Because that situation can prevent the awsdms_intercept_ddl event trigger from firing. To work around the situation, modify that trigger on your source database as follows:

alter event trigger awsdms_intercept_ddl enable always;
AWS
beantwortet vor 2 Monaten
profile picture
EXPERTE
überprüft vor 2 Monaten
  • After executing the query "alter event trigger awsdms_intercept_ddl enable always;" on the source database, I observed that newly created tables were successfully transformed. However, going forward, we're exploring ways to automate this process without the need to manually enable event triggers. how can i?

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen