DynamoDB streams to EventBridge Pipe Start/Stop

Question

If I have a working pipeline with DDB streams and a Pipe, then I stop the pipe and start it again (10 minutes) any modification in DDB during the stopped time will be processed by the Pipe when it started again.

Is it possible to change this behavior? Doesnt pool the messages while the Pipe is stopped?

Accepted Answer

Event Bridge Pipes and Lambda's Event Source Mapping work the same way, if you simply stop the mapping, it retains a pointer to the last iterator position and knows how to pick up where it left off.

However, you wish for it to skip any events in between, which means you will need to do 1 of two things:

1. Delete the pipe, and re-create it as opposed to stopping it.
2. If you have timestamps in your items, you can add a filter to your pipe, to not invoke within a given time range.

Answer

Hi, EventBridge Pipes can be configured to work in batch mode where it buffers events up to five minutes (or 6 MB)

See https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-dynamodb.html

```
To avoid processing a small number of records, you can tell the pipe to buffer 
records for up to five minutes by configuring a batching window. Before processing 
the events, EventBridge continues to read records from the source until it has gathered 
a full batch, the batching window expires, or the batch reaches the payload limit of 6 MB.
```

If this doesn't solve your use case, you will have to go to some buffering mechanism on your own: never stop the pipe, get all events stored in a DDB table (or S3 bucket) that you manage and re-stream on a Pipe managed by you those when consumer is ready to accept them.

I'd also suggest to read this blog post, whose mechanisms are (partially) applicable to what you're trying to achieve.

Best,

Didier

DynamoDB streams to EventBridge Pipe Start/Stop

関連するコンテンツ