Skip to content

Add AWS Salesforce Zero ETL All tables processed event.

0

I want to Trigger a lambda Function when my Zero-ETL Salesforce integration complete for all objects. This lambda function will trigger a dbt workflow in github that will use the synced data in Snowflake with an external table.

Right now the Eventbridge rule is triggered for individual objects, is there a way we could get "Job complete all objects processed" as an eventbridge event from aws.glue-zero-etl source?

There is an object level event, but I want an overall event to fire after the last table is checked.

https://aws.amazon.com/blogs/big-data/accelerate-aws-glue-zero-etl-data-ingestion-using-salesforce-bulk-api/

1 Answer
1

Based on the current AWS Glue zero-ETL integration capabilities, there isn't a built-in "all tables processed" or "job complete" event available from the aws.glue-zero-etl source in EventBridge. The zero-ETL integration events are emitted at the individual entity (table/object) level.

The available EventBridge events for AWS Glue zero-ETL integrations include:

  • AWS Glue Zero ETL Ingestion Completed (per entity)
  • AWS Glue Zero ETL Ingestion Failed (per entity)
  • AWS Glue Zero ETL Integration Resynced
  • AWS Glue Zero ETL Integration Failed
  • AWS Glue Zero ETL Integration Needs Attention
  • AWS Glue Zero ETL Ingestion In Progress (per entity)

To work around this limitation and trigger your Lambda function after all objects are processed, you could implement a solution where:

  1. Your EventBridge rule captures the individual "Ingestion Completed" events for each object
  2. Your Lambda function tracks which objects have completed (perhaps using DynamoDB or another state store)
  3. When the Lambda detects that all expected objects have been processed, it then triggers your dbt workflow in GitHub

Alternatively, you could monitor the CloudWatch metrics that are emitted for zero-ETL integrations, specifically the LastSyncTimestamp metric, and use CloudWatch alarms or scheduled checks to determine when all tables have been synced before triggering your downstream workflow.

I recommend providing feedback to AWS through your AWS Support case or account team about the need for an aggregate "all objects processed" event, as this would be a valuable enhancement for orchestrating downstream workflows.
Sources
Monitoring an integration - AWS Glue

answered 2 months ago

EXPERT

reviewed 2 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.