Glue incremental load

0

I am loading the data from Amazon RDS(mysql database) to Redshift using AWS Glue ETL and data catolgue. But I can't figure out how to do incremental loading(upsert)? Is there a way to create a filter/parameter on date while reading from source database to load only yesterday's data without using bookmarks?

2 回答
0

Using Bookmarks is recommended way when using Glue, however nothing stops you from passing a parameter to glue script [https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-crawler-pyspark-extensions-get-resolved-options.html] and then generaly the query dynamically based on that parameter if you want.

The other question is why you wanted to use Glue and not Redshift federated query instead which might be an easier option?

AWS
Alex_T
已回答 1 年前
  • Cause i use Redshift Serverless which still have no Query schedule , so even if i used federated query to run MV i will have to schedule using step function to be able to refresh and monitor extraction workflow.

    the issue of using the bookmark that i need only the data that is loaded yesterday and by default running the jobs after 12 AM will cause some unneeded data to be loaded by using bookmark which might affect my extraction logic

0

One option is to use DMS to incrementally load Redshift using MySQL as a source. Here's a whitepaper on the topic: https://docs.aws.amazon.com/whitepapers/latest/optimizing-dms-with-amazon-redshift/optimizing-dms-with-amazon-redshift.html

profile pictureAWS
专家
Rajiv_G
已回答 1 年前

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。

回答问题的准则