Recommendation: AWS Batch or Glue and need information to decide which machine to choose

Question

My company is pushing to have everything "As Code" and I was wondering which tool is better for us.
Since we are not allowed to use the Glue interface to create ETLs, to configure the settings, and need to write everything in python then push via Jenkins/whatever, is there any advantages for us to choose Glue?

Also, if I use Batch, do I have to pay for another machine (on demand) to run my ETLs or can I use the resources of my RDS instance for PostgreSQL for my ETLs?

Answer

Glue provides both a visual interface through Glue Studio as well as the ability to write Python/Scala scripts  taking advantage of Glue's capabilities like catalog integration and managed execution.

Batch on the other hand requires writing and managing your own ETL code without the catalog integration that Glue provides. You would need to provision EC2 capacity to run Batch jobs.

Both Glue and Batch jobs can be triggered on a schedule or in response to events. For Batch, you may want to look at using AWS Step Functions to orchestrate and monitor the jobs.

Recommendation: AWS Batch or Glue and need information to decide which machine to choose

Contenuto pertinente