- Newest
- Most votes
- Most comments
Based on the current AWS Glue zero-ETL integration capabilities, there isn't a built-in "all tables processed" or "job complete" event available from the aws.glue-zero-etl source in EventBridge. The zero-ETL integration events are emitted at the individual entity (table/object) level.
The available EventBridge events for AWS Glue zero-ETL integrations include:
- AWS Glue Zero ETL Ingestion Completed (per entity)
- AWS Glue Zero ETL Ingestion Failed (per entity)
- AWS Glue Zero ETL Integration Resynced
- AWS Glue Zero ETL Integration Failed
- AWS Glue Zero ETL Integration Needs Attention
- AWS Glue Zero ETL Ingestion In Progress (per entity)
To work around this limitation and trigger your Lambda function after all objects are processed, you could implement a solution where:
- Your EventBridge rule captures the individual "Ingestion Completed" events for each object
- Your Lambda function tracks which objects have completed (perhaps using DynamoDB or another state store)
- When the Lambda detects that all expected objects have been processed, it then triggers your dbt workflow in GitHub
Alternatively, you could monitor the CloudWatch metrics that are emitted for zero-ETL integrations, specifically the LastSyncTimestamp metric, and use CloudWatch alarms or scheduled checks to determine when all tables have been synced before triggering your downstream workflow.
I recommend providing feedback to AWS through your AWS Support case or account team about the need for an aggregate "all objects processed" event, as this would be a valuable enhancement for orchestrating downstream workflows.
Sources
Monitoring an integration - AWS Glue
Relevant content
asked 2 years ago
