Redshift Serverless Integration with ZeroETL OR FDW

0

Good day,

  1. We setting up Redshiift serverless, due to not having impact on production RDS sources we looking at zeroetl or fdw with CDC, does zeroetl replication incur cost on serverless as a transaction is created (Near realtime) it moves across to the external schema on serverless as it removes the pause state? Management is not keen on batch process due to impact on prod, or do i use FDW cdc? (how)

  2. S3 data transfer NRT do i use Redshift spectrum to integrate to serverless?

  3. RDS MS-SQL integrate to Redsdift serverless, DMS or ?

  4. Job scheduling for some API call do i use redshift schedular (can it do api call) or MWAA, i like one view to manage?

Please Help Regards

IC
已提問 2 個月前檢視次數 397 次
3 個答案
1
已接受的答案

Hello , Please find my response inline

We setting up Redshiift serverless, due to not having impact on production RDS sources we looking at zeroetl or fdw with CDC, does zeroetl replication incur cost on serverless as a transaction is created (Near realtime) it moves across to the external schema on serverless as it removes the pause state? Management is not keen on batch process due to impact on prod, or do i use FDW cdc? (how) Answer - Need to check on the pricing.

S3 data transfer NRT do i use Redshift spectrum to integrate to serverless?

Answer - You dont need to move data to Redshift from S3 , you can use Redshift spectrum to read data from S3 for various file formats. If you need to store the S3 data locally then you can do so by inserting into Redshift local table from S3 data using Redshift spectrum. https://docs.aws.amazon.com/redshift/latest/dg/r_INSERT_external_table.html

RDS MS-SQL integrate to Redsdift serverless, DMS or ?

Answer - You can look into ZeroETL which is in preview currently to do the same CDC from RDS MySQL to Redshift simplifying the pipeline. If you need to implement the CDC in production now, you can use DMS to do so.

Job scheduling for some API call do i use redshift schedular (can it do api call) or MWAA, i like one view to manage?

Answer - You can use Redshift Query Editor V2 to schedule the queries/jobs from the console. If you are looking for at orchestration you can look at using Step functions leveraging Redshift Data API. MWAA is also another option to use for orchestration and scheduling Redshift queries. It depends on which solution is cost effective and easier for you to adopt. https://aws.amazon.com/blogs/big-data/etl-orchestration-using-the-amazon-redshift-data-api-and-aws-step-functions-with-aws-sdk-integration/

AWS
專家
已回答 2 個月前
profile picture
專家
已審閱 2 個月前
0

Good day,

Please help, I'm a bit lost on the best method going forward Any input is much appreciated..

IC
已回答 2 個月前
0

Thank You very much for sharing your knowledge, I think I am on the correct path.

IC
已回答 2 個月前

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南