CDC files not getting generated when replicating data from oracle RDS to s3.

Question

We are trying to Replicate Data from Oracle RDS to Target oracle RDS by Masking PII data For that we are following below approach:

1. Replicating data from Oracle RDS to 53 bucket by masking PII data and then Transferring data from s3 bucket to target RDS by renaming Column name back to original name and removing first column which has operation details like 'I' etc..

But when following above approach we observed CDC files are not getting generated when Transferring Data from RDS to s3 and getting error like CDC failed to get table definition: . In source endpoint we have specified "cdcPath" in endpoint settings as well?

Can someone please help us with above issue?
Below are the mapping rules which we used to transfer from source to s3:

{
"rules":[
"rule-type": "selection",
"rule-id": "1",
"rule-name": "1",
"object-locator": {
"schema-name": "OWN",
"table-name": "CONTENT"
},
"rule-action": "include",
"filters": [],
"parallel-load": null,
"isAutoSegmentationChecked": false
},
{
"rule-type": "transformation",
"rule-id": "2",
"rule-name": "adcolpass",
"rule-target": "column",
"object-locator": {
"schema-name": "OWN",
"table-name": "CONTENT"
},
"rule-action": "add-column",
"value": "DMS_ID",
"expression": "C"||hash_sha256($ID)",
"data-type": {
"type": "string",
"length": 100
},
"old-value": null
},
{
"rule-type": "transformation",
"rule-id": "3",
"rule-name": "3",
"rule-target": "column",
"object-locator": {
"schema-name": "OWN",
"table-name": "CONTENT"
},
{
"rule-action": "add-column",
"value": "DMS_DISPLAYNAME",
"expression": ""d"||hash_sha256($DISPLAY_NAME)",
"data-type": {
"type": "string",
"length": 100
},
"old-value": null
},
{
"rule-type": "transformation",
"rule-id": "4",
"rule-name": "4",
"rule-target": "column",
"object-locator": {
"schema-name": "OWN",
"table-name": "CONTENT",
"column-name": "ID"
},
"rule-action": "remove-column",
"value": null,
"old-value": null
},
{
"rule-type": "transformation",
"rule-id": "5",
"rule-name": "5",
"rule-target": "column",
"object-locator": {
"schema-name": "OWN",
"table-name": "CONTENT",
"column-name": "DISPLAY_NAME"
},
"rule-action": "remove-column",
"value": null,
"old-value": null
}
]
}

Mapping rules used to transfer from s3 to target RDS:

{
"rules":[
"rule-type": "selection",
"rule-id": "1",
"rule-name": "1",
"object-locator": {
"schema-name": "%",
"table-name": "%"
},
"rule-action": "include",
"filters":[],
"parallel-load": null,
"isAutoSegmentationChecked": false
},
{
"rule-type": "transformation",
"rule-id": "2",
"rule-name": "2",
"rule-target": "column",
"object-locator": {
"schema-name": "OWN",
"table-name": "CONTENT",
"column-name": "Op"
},
"rule-action": "remove-column"
},
{
"rule-type": "transformation",
"rule-id": "3",
"rule-name": "3",
"rule-action": "rename",
"rule-target":"column",
"object-locator": {
"schema-name": "OWN",
"table-name": "CONTENT",
"column-name": "DMS_DISPLAYNAME"
},
"value": "DISPLAY_NAME"
},
{
"rule-type": "transformation",
"rule-id": "4",
"rule-name": "4",
"rule-action": "rename",
"rule-target": "column",
"object-locator": { 
"schema-name": "OWN", 
"table-name": "CONTENT", 
"column-name": "DMS_ID" 
},
"value": "ID"
}
]
}

Answer

This error can occur if the S3 CSV file formatting is incorrect. The CSV file should be formatted as follows [1]:

Operation – the change operation to be performed: INSERT or I, UPDATE or U, or DELETE or D. These keyword and character values are case-insensitive.
Table name – the name of the source table.
Schema name – the name of the source schema.
Data – one or more columns that represent the data to be changed.

This is an example of the formatting for data within the CSV file:

INSERT,employee,hr,101,Smith,Bob,2014-06-04,New York
UPDATE,employee,hr,101,Smith,Bob,2015-10-08,Los Angeles
UPDATE,employee,hr,101,Smith,Bob,2017-03-13,Dallas
DELETE,employee,hr,101,Smith,Bob,2017-03-13,Dallas

CDC files not getting generated when replicating data from oracle RDS to s3.

相關內容