By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

How to achieve data change capture using AWS Glue and RDS oracle

0

I'm trying to achieve data change capture using AWS Glue and don't want to use DMS. I'm trying to transfer data between two Oracle RDS instances which are in different AWS Account. Here I am trying to follow Delta approach (i.e., I am to transfer only updated data...) From source, we are going to fetch records for which updated data of record is greater than last job run date and place all those records in Delta table and from Delta table, we wanted to manually upsert the records to target table based on primary key. Currently we are using cursors to do upsert operation, but it seems requires cx_Oracle or oracle libraries to connect to oracle. Those libraries need to be added into AWS Glue job externally, but we are facing multiple issues while doing so.

Can anyone please suggest us the way to upsert the records to target RDS from delta tables using any straightforward approach which AWS Glue provides?

Topics
AnalyticsMigration & ModernizationDatabase
Tags
AnalyticsAWS Database Migration ServiceAmazon Relational Database ServiceAWS GlueExtract Transform & Load Data
Language
English
Abhijeet
asked 4 months ago442 views
1 Answer
2

For Oracle, Glue doesn't provide upsert out of the box, you need to write to a temporary table and then run the upsert into the final table using a SQL command (via JDBC or oracledb)

profile pictureAWS
EXPERT
Gonzalo Herreros
answered 3 months ago
  • Abhijeet
    3 months ago

    I have scenario to dealt with replicate data around multiple tables. In this case, we would ended up creating multiple temporary tables instead do we have any simplified approach to avoid multiple temporary tables creation? There is a definite possibility of data volume keep growing in some of the tables. Is AWS Glue is providing any solid approach in case of storing/transferring large datasets?

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content