Fanning out DynamoDB stream into a kinesis stream using a lambda trigger?

0

Hi, I'm trying to implement a system which essentially provides in-order item-level information about DDB updates to multiple consumers. I'm using DDB streams as Kinesis streams from DDB don't maintain order and deduping (source). However, DDB streams can only have a max. of 2 concurrent consumers before being throttled.

So what I'm trying to do is have the DDB stream trigger a lambda, which serializes the com.amazonaws.services.lambda.runtime.events.DyanmodbEvent.DynamodbStreamRecord, and passes it into a kinesis stream. A few questions:

  1. What's the best method of serializing the DynamodbStreamRecord? I've seen some examples using KCL and the RecordAdapter, but that operates on the com.amazonaws.services.dynamodbv2.model.Record object, and not the lambda event objects DynamodbStreamRecord and StreamRecord.
  2. When I'm writing code to send data into a kinesis stream, the putRecord API in the kinesis client requires a partition key parameter. If I'm aiming to maintain the same order on an item level in the kinesis stream, should the partition key that I supply for the putRecord call just be the partition key of the relevant item? Thanks!
1 Answer
3
Accepted Answer

Hi,

may not be a direct answer, but consider whether you could use EventBridge Pipes with DynamoStream as input instead: https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-dynamodb.html.

From there, you could have the following targets as choice: https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-event-target.html.

I mention this option because I have experience that having single lambda as "manager" of the dynamo stream, will result in potential throttles and costs depending on the data traffic.

Hope it inspires you ;)

profile picture
EXPERT
answered 2 years ago
profile picture
EXPERT
reviewed 6 months ago
profile picture
EXPERT
reviewed 7 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions