Glue Workflow: Add same glue job multiple times, each with different parameters

Hello,

To achieve your desired workflow of running the same Glue job multiple times with different parameters for each execution, you can follow these steps:

Create a Parameterised Glue Job[1]: First, make sure your Glue job is parameterised properly so that you can pass different values for database, table, and destination bucket each time it runs. You can use Glue's Job Parameters feature to achieve this. When defining your Glue job, declare the parameters you want to be able to pass dynamically, like database, table, and destination_bucket. Please make sure that the role making the call to S3 has the required S3 permissions (assuming this operation is on the same account).

Use AWS Step Functions: AWS Step Functions allow you to create a workflow that coordinates multiple AWS services, including Glue jobs. You can use Step Functions to loop through a list of tables and invoke the same Glue job multiple times, each time with different parameters. Here is a link on how you can create a step function workflow. Link URL: https://docs.aws.amazon.com/step-functions/latest/dg/tutorial-workflow-studio-using.html

References: [1]AWS Glue job parameters https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html

[2]What is AWS Step Functions? https://docs.aws.amazon.com/step-functions/latest/dg/welcome.html