AWS Glue create 2 Triggers in one workflow from Boto3

0

Can we add two triggers to same workflow of each trigger with different type. One with CONDITIONAL and Other trigger with SCHEDULED

My workflow design is There is SCHEDULED Trigger with two jobs . On Successful Completion on these two Jobs there has to be invocation of one more trigger.

Unfortunately I could not find any documentation in various online forums. Can anyone help here please

asked a year ago880 views
2 Answers
2

yes, this is possible. I have created a sampe below for you to reference

  • The start trigger is a scheduled trigger. This triggers 2 jobs (GlueJob and JDBCETLGlueJob in the picture)
  • There is conditional trigger that you added after both jobs complete. the option is to start after "ALL" requirements are met - i.e. both jobs complete successfully.
  • The conditional trigger starts another Glue Job

Enter image description here

profile pictureAWS
answered a year ago
  • This is exactly what I need. Thank you Ananth . How do I code using boto3 ?. It is allowing me to define action and type only once

1

If this helps, please accept answer and upvote so that it helps others. Here's an example of a Python program that uses the Boto3 library to create a Glue trigger that triggers "glue-job-1" and "glue-job-2", and another conditional trigger that triggers "glue-job-3" when both "glue-job-1" and "glue-job-2" complete successfully:

import boto3

# Create a Glue client
glue_client = boto3.client('glue')

# Define the names of the jobs to trigger
job_1_name = "glue-job-1"
job_2_name = "glue-job-2"
job_3_name = "glue-job-3"

# Define the trigger that triggers job_1 and job_2
job_1_2_trigger = {
    "Name": "job_1_2_trigger",
    "Type": "ON_DEMAND",
    "Actions": [
        {
            "JobName": job_1_name
        },
        {
            "JobName": job_2_name
        }
    ]
}

# Create the trigger
response = glue_client.create_trigger(Trigger=job_1_2_trigger)
print(f"Successfully created trigger {response['Name']}")

# Define the condition for the second trigger
condition = f"SUCCEEDED({job_1_name}) AND SUCCEEDED({job_2_name})"

# Define the trigger that triggers job_3
job_3_trigger = {
    "Name": "job_3_trigger",
    "Type": "CONDITIONAL",
    "Actions": [
        {
            "JobName": job_3_name
        }
    ],
    "Predicate": {
        "Conditions": [
            {
                "LogicalOperator": "AND",
                "JobName": condition
            }
        ]
    }
}

# Create the trigger
response = glue_client.create_trigger(Trigger=job_3_trigger)
print(f"Successfully created trigger {response['Name']}")

This program uses the glue_client.create_trigger() method to create the triggers and the glue_client.create_trigger() method to create the triggers. The job_1_2_trigger is of type ON_DEMAND that triggers job_1 and job_2 . The job_3_trigger is of type CONDITIONAL that triggers job_3 when both job_1 and job_2 complete successfully.

You would need to add appropriate error handling, input validation and other requirements specific to your use case.

profile pictureAWS
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions