AWS Glue create 2 Triggers in one workflow from Boto3

0

Can we add two triggers to same workflow of each trigger with different type. One with CONDITIONAL and Other trigger with SCHEDULED

My workflow design is There is SCHEDULED Trigger with two jobs . On Successful Completion on these two Jobs there has to be invocation of one more trigger.

Unfortunately I could not find any documentation in various online forums. Can anyone help here please

已提問 1 年前檢視次數 896 次
2 個答案
2

yes, this is possible. I have created a sampe below for you to reference

  • The start trigger is a scheduled trigger. This triggers 2 jobs (GlueJob and JDBCETLGlueJob in the picture)
  • There is conditional trigger that you added after both jobs complete. the option is to start after "ALL" requirements are met - i.e. both jobs complete successfully.
  • The conditional trigger starts another Glue Job

Enter image description here

profile pictureAWS
已回答 1 年前
  • This is exactly what I need. Thank you Ananth . How do I code using boto3 ?. It is allowing me to define action and type only once

1

If this helps, please accept answer and upvote so that it helps others. Here's an example of a Python program that uses the Boto3 library to create a Glue trigger that triggers "glue-job-1" and "glue-job-2", and another conditional trigger that triggers "glue-job-3" when both "glue-job-1" and "glue-job-2" complete successfully:

import boto3

# Create a Glue client
glue_client = boto3.client('glue')

# Define the names of the jobs to trigger
job_1_name = "glue-job-1"
job_2_name = "glue-job-2"
job_3_name = "glue-job-3"

# Define the trigger that triggers job_1 and job_2
job_1_2_trigger = {
    "Name": "job_1_2_trigger",
    "Type": "ON_DEMAND",
    "Actions": [
        {
            "JobName": job_1_name
        },
        {
            "JobName": job_2_name
        }
    ]
}

# Create the trigger
response = glue_client.create_trigger(Trigger=job_1_2_trigger)
print(f"Successfully created trigger {response['Name']}")

# Define the condition for the second trigger
condition = f"SUCCEEDED({job_1_name}) AND SUCCEEDED({job_2_name})"

# Define the trigger that triggers job_3
job_3_trigger = {
    "Name": "job_3_trigger",
    "Type": "CONDITIONAL",
    "Actions": [
        {
            "JobName": job_3_name
        }
    ],
    "Predicate": {
        "Conditions": [
            {
                "LogicalOperator": "AND",
                "JobName": condition
            }
        ]
    }
}

# Create the trigger
response = glue_client.create_trigger(Trigger=job_3_trigger)
print(f"Successfully created trigger {response['Name']}")

This program uses the glue_client.create_trigger() method to create the triggers and the glue_client.create_trigger() method to create the triggers. The job_1_2_trigger is of type ON_DEMAND that triggers job_1 and job_2 . The job_3_trigger is of type CONDITIONAL that triggers job_3 when both job_1 and job_2 complete successfully.

You would need to add appropriate error handling, input validation and other requirements specific to your use case.

profile pictureAWS
已回答 1 年前

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南