I'm trying to achieve data change capture using AWS Glue and don't want to use DMS. I'm trying to transfer data between two Oracle RDS instances which are in different AWS Account. Here I am trying to follow Delta approach (i.e., I am to transfer only updated data...)
From source, we are going to fetch records for which updated data of record is greater than last job run date and place all those records in Delta table and from Delta table, we wanted to manually upsert the records to target table based on primary key.
Currently we are using cursors to do upsert operation, but it seems requires cx_Oracle or oracle libraries to connect to oracle. Those libraries need to be added into AWS Glue job externally, but we are facing multiple issues while doing so.
Can anyone please suggest us the way to upsert the records to target RDS from delta tables using any straightforward approach which AWS Glue provides?
AWS UFFICIALEAggiornata 8 mesi fa
I have scenario to dealt with replicate data around multiple tables. In this case, we would ended up creating multiple temporary tables instead do we have any simplified approach to avoid multiple temporary tables creation? There is a definite possibility of data volume keep growing in some of the tables. Is AWS Glue is providing any solid approach in case of storing/transferring large datasets?