AWS Glue Workflow Trigger

0

I have a glue job (job_a) that starts through a Lambda. When a file is placed inside an S3 bucket, I am triggering a glue job (job_a) through Lambda. My requirement is, once this glue job (job_a), is completed SUCCESSFULLY, I need to start another glue job (job_b). I created a trigger where target resource is job_b and watched resource is job_a. This trigger is added into a glue workflow. But the workflow is not getting started even if job_a is completed SUCCESS. What am i missing here? Do I need to include a EventBrdige here? Can someone help me on this?

1 Antwort
0

To trigger job_b after job_a completes successfully using Glue workflows:

  • Create a Glue workflow with two triggers - one each for job_a and job_b
  • The trigger for job_a should have job_a as the target resource
  • The trigger for job_b should have job_b as the target resource and job_a as the watched resource
  • This will make the trigger for job_b fire only after job_a is completed successfully
  • Add both triggers to the Glue workflow
  • Start job_a through any method like Lambda. The workflow will take care of automatically starting job_b after job_a is done

You do not need EventBridge for this use case as Glue workflows are designed to handle these dependencies between jobs.

profile picture
EXPERTE
beantwortet vor 2 Monaten
  • Thanks for your reply. I tried doing this, but did not work. As you have mentioned in the 1st point, I created a trigger with "Trigger Type" as "Job Or Crawler Event" and "Conditional Logic" to "All". The I gave watched and target resource as JOB_A and tried creating it. But the trigger failed to create with error: "error message: Job cannot be a Predicate and Action for Conditional Trigger.". We cannot trigger with watched & target event as same job. Also cannot skip the "Watched Resource" option. Can you please check this? Not sure, If I misunderstood what you told or missing something here?

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen