2 Respuestas
- Más nuevo
- Más votos
- Más comentarios
1
There is no native option to pass EventBridge event details to Glue job. But I used the following workaround to do this:
- You need to get an event ID from Glue workflow properties
event_id = glue_client.get_workflow_run_properties(Name=self.args['WORKFLOW_NAME'],
RunId=self.args['WORKFLOW_RUN_ID'])['RunProperties']['aws:eventIds'][1:-1]
- Get all NotifyEvent events for the last several minutes. It's up to you to decide how much time can pass between the workflow start and your job start.
response = event_client.lookup_events(LookupAttributes=[{'AttributeKey': 'EventName',
'AttributeValue': 'NotifyEvent'}],
StartTime=(datetime.datetime.now() - datetime.timedelta(minutes=5)),
EndTime=datetime.datetime.now())['Events']
- Check which event has an enclosed event with the event ID we get from Glue workflow.
for i in range(len(response)):
event_payload = json.loads(response[i]['CloudTrailEvent'])['requestParameters']['eventPayload']
if event_payload['eventId'] == event_id:
event = json.loads(event_payload['eventBody'])
In the event variable you get the full content of the event that triggered Glue workflow.
respondido hace un año
0
To pass a parameter to a workflow, use StartWorkflowRun[1] API with RunProperties which takes key-value pairs and it can be accessed in the any job in the workflow using GetWorkflowRunProperties[2]. Further, if required, it can be modified in any job in the workflow using PutWorkflowRunProperties[3].
If triggering an ETL job instead of a workflow, use StartJobRun[4] and set the job arguments. To access these job arguments in the script, use getResolvedOptions[5]
respondido hace 2 años
Contenido relevante
- ¿Cómo superviso los eventos de IAM y configuro las notificaciones de eventos con AWS CloudFormation?OFICIAL DE AWSActualizada hace 2 años
- OFICIAL DE AWSActualizada hace 8 meses
Hi Sachin,
The one part that isn't clear to me from your answer is if I have a trigger configured in glue based on an EventBridge event, how would you map the event contents to the job as params? The example in the question is the field database_name. If that value can be different for each event how do you configure the trigger to take that field and pass it in as a key \ value? If you can't configure it in the trigger is there a way once the job starts to get the event body in the job to extract the info?