Redshift Serverless Integration with ZeroETL OR FDW

0

Good day,

  1. We setting up Redshiift serverless, due to not having impact on production RDS sources we looking at zeroetl or fdw with CDC, does zeroetl replication incur cost on serverless as a transaction is created (Near realtime) it moves across to the external schema on serverless as it removes the pause state? Management is not keen on batch process due to impact on prod, or do i use FDW cdc? (how)

  2. S3 data transfer NRT do i use Redshift spectrum to integrate to serverless?

  3. RDS MS-SQL integrate to Redsdift serverless, DMS or ?

  4. Job scheduling for some API call do i use redshift schedular (can it do api call) or MWAA, i like one view to manage?

Please Help Regards

3 Answers
1
Accepted Answer

Hello , Please find my response inline

We setting up Redshiift serverless, due to not having impact on production RDS sources we looking at zeroetl or fdw with CDC, does zeroetl replication incur cost on serverless as a transaction is created (Near realtime) it moves across to the external schema on serverless as it removes the pause state? Management is not keen on batch process due to impact on prod, or do i use FDW cdc? (how) Answer - Need to check on the pricing.

S3 data transfer NRT do i use Redshift spectrum to integrate to serverless?

Answer - You dont need to move data to Redshift from S3 , you can use Redshift spectrum to read data from S3 for various file formats. If you need to store the S3 data locally then you can do so by inserting into Redshift local table from S3 data using Redshift spectrum. https://docs.aws.amazon.com/redshift/latest/dg/r_INSERT_external_table.html

RDS MS-SQL integrate to Redsdift serverless, DMS or ?

Answer - You can look into ZeroETL which is in preview currently to do the same CDC from RDS MySQL to Redshift simplifying the pipeline. If you need to implement the CDC in production now, you can use DMS to do so.

Job scheduling for some API call do i use redshift schedular (can it do api call) or MWAA, i like one view to manage?

Answer - You can use Redshift Query Editor V2 to schedule the queries/jobs from the console. If you are looking for at orchestration you can look at using Step functions leveraging Redshift Data API. MWAA is also another option to use for orchestration and scheduling Redshift queries. It depends on which solution is cost effective and easier for you to adopt. https://aws.amazon.com/blogs/big-data/etl-orchestration-using-the-amazon-redshift-data-api-and-aws-step-functions-with-aws-sdk-integration/

AWS
EXPERT
answered 2 months ago
profile picture
EXPERT
reviewed 2 months ago
0

Good day,

Please help, I'm a bit lost on the best method going forward Any input is much appreciated..

IC
answered 3 months ago
0

Thank You very much for sharing your knowledge, I think I am on the correct path.

IC
answered 2 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions