What is the best ETL tool for ongoing loads of data into Redshift?

Question

What would be the best AWS ETL tool that a customer would use to set up ongoing loads of data into Redshift, that would provide some sort of transform functionality, similar to Microsoft SSIS? E.g. “load data from this file into this table daily as a full replace, compute these columns, etc.”

Accepted Answer

Depending on the customer's anticipated timeframe, this is precisely what Glue (https://aws.amazon.com/glue/ ) is intended to be really good at.

Alternatively, if your customer can persuade their data source to stream its content (what is their data source, btw?), Kinesis with its capability to trigger Lambda functions for the "TL" piece, may be a good fit - see eg https://aws.amazon.com/blogs/big-data/tag/aws-lambda/ .

Answer

Is it possible to run an advanced SQL query on the Glue job? I have at least 15 tables in my SQL and the query is quite advanced itself. Doesn't Glue work only with a small number of tables like e.g. 1-3 with simple conditions? Is there an option to run my own query on it, without building the query by using boxes in Job?

What is the best ETL tool for ongoing loads of data into Redshift?

Relevant content