By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

AWS Glue Workflow Trigger

0

I have a glue job (job_a) that starts through a Lambda. When a file is placed inside an S3 bucket, I am triggering a glue job (job_a) through Lambda. My requirement is, once this glue job (job_a), is completed SUCCESSFULLY, I need to start another glue job (job_b). I created a trigger where target resource is job_b and watched resource is job_a. This trigger is added into a glue workflow. But the workflow is not getting started even if job_a is completed SUCCESS. What am i missing here? Do I need to include a EventBrdige here? Can someone help me on this?

Topics
ServerlessComputeManagement & GovernanceApplication IntegrationAnalyticsDatabase
Tags
AWS LambdaAWS CloudFormationAmazon EventBridgeAWS GlueExtract Transform & Load Data
Language
English
Joe
asked 2 months ago301 views
1 Answer
0

To trigger job_b after job_a completes successfully using Glue workflows:

  • Create a Glue workflow with two triggers - one each for job_a and job_b
  • The trigger for job_a should have job_a as the target resource
  • The trigger for job_b should have job_b as the target resource and job_a as the watched resource
  • This will make the trigger for job_b fire only after job_a is completed successfully
  • Add both triggers to the Glue workflow
  • Start job_a through any method like Lambda. The workflow will take care of automatically starting job_b after job_a is done

You do not need EventBridge for this use case as Glue workflows are designed to handle these dependencies between jobs.

profile picture
EXPERT
Giovanni Lauria
answered 2 months ago
  • Joe
    2 months ago

    Thanks for your reply. I tried doing this, but did not work. As you have mentioned in the 1st point, I created a trigger with "Trigger Type" as "Job Or Crawler Event" and "Conditional Logic" to "All". The I gave watched and target resource as JOB_A and tried creating it. But the trigger failed to create with error: "error message: Job cannot be a Predicate and Action for Conditional Trigger.". We cannot trigger with watched & target event as same job. Also cannot skip the "Watched Resource" option. Can you please check this? Not sure, If I misunderstood what you told or missing something here?

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content