Help creating Source to Target Mapping Report using Glue that describes the mapping and behavior of existing Glue Jobs.

Question

I am using glue to move data from s3 and redshift into a data lake. I would like to use a combination of AWS glue and Athena to create a source to target mapping report. The goal is to make the report dynamic, so that if an element in the ETL process is changed (i.e. a table name is altered) the report will adjust accordingly to the respective changes that have been made. What is the best practice that has been used to complete this task? Or what solutions exist that help solve this problem.

Answer

If it's just the table name, just a job argument and then use the argument in the code.  
For something more sophisticated you probably need some parameters and code logic, or just build a SparkSQL query dynamically.

It's not common practice to combine Glue and Athena for the same task (e.g. mapping and saving the data), in this case it sounds like Glue is more suited for that ETL.

Help creating Source to Target Mapping Report using Glue that describes the mapping and behavior of existing Glue Jobs.

相關內容