2 Answers
- Newest
- Most votes
- Most comments
1
There is no native option to pass EventBridge event details to Glue job. But I used the following workaround to do this:
- You need to get an event ID from Glue workflow properties
event_id = glue_client.get_workflow_run_properties(Name=self.args['WORKFLOW_NAME'],
RunId=self.args['WORKFLOW_RUN_ID'])['RunProperties']['aws:eventIds'][1:-1]
- Get all NotifyEvent events for the last several minutes. It's up to you to decide how much time can pass between the workflow start and your job start.
response = event_client.lookup_events(LookupAttributes=[{'AttributeKey': 'EventName',
'AttributeValue': 'NotifyEvent'}],
StartTime=(datetime.datetime.now() - datetime.timedelta(minutes=5)),
EndTime=datetime.datetime.now())['Events']
- Check which event has an enclosed event with the event ID we get from Glue workflow.
for i in range(len(response)):
event_payload = json.loads(response[i]['CloudTrailEvent'])['requestParameters']['eventPayload']
if event_payload['eventId'] == event_id:
event = json.loads(event_payload['eventBody'])
In the event variable you get the full content of the event that triggered Glue workflow.
answered a year ago
0
To pass a parameter to a workflow, use StartWorkflowRun[1] API with RunProperties which takes key-value pairs and it can be accessed in the any job in the workflow using GetWorkflowRunProperties[2]. Further, if required, it can be modified in any job in the workflow using PutWorkflowRunProperties[3].
If triggering an ETL job instead of a workflow, use StartJobRun[4] and set the job arguments. To access these job arguments in the script, use getResolvedOptions[5]
answered 2 years ago
Relevant content
- asked 2 years ago
- asked 7 months ago
- asked 9 months ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated a year ago
Hi Sachin,
The one part that isn't clear to me from your answer is if I have a trigger configured in glue based on an EventBridge event, how would you map the event contents to the job as params? The example in the question is the field database_name. If that value can be different for each event how do you configure the trigger to take that field and pass it in as a key \ value? If you can't configure it in the trigger is there a way once the job starts to get the event body in the job to extract the info?