1 Answer
- Newest
- Most votes
- Most comments
0
AWS Step Functions provides a great way of orchestrating multiple Glue jobs into a coherent workflow.
Here is a workshop that shows an example on how to build a workflow using Step Functions. It provides a visual interface and can also be defined programmatically using the Amazon States Language.
You can store the state of the different Glue jobs (as flags) in DynamoDB so you can build a fully serverless data pipelines. (Glue, Step Functions and DynamoDB are all serverless).
You can also consider event-driven orchestration of the different workflows using Amazon EventBridge.
answered 2 years ago
Relevant content
- asked 3 months ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 3 years ago
- AWS OFFICIALUpdated a year ago