Suggest a failover mechanism when replicating from DynamoDb Table to OpenSearch

Question

I integrated Change Data Capture into an OpenSearch index using DynamoDB Streams and a Lambda Trigger. Now, I'm grappling with the challenge of managing potential disruptions in the availability of the OpenSearch cluster. My goal is to establish a robust solution involving exponential backoff in case of failover. I'm concerned about the cost-effectiveness of implementing and retrying logic directly within Lambda. 
And using built-in retries didn't introduce any delay between each retry.

Unfortunately, utilizing a First-In-First-Out (FIFO) Amazon Simple Queue Service (SQS) queue as a Dead Letter Queue through DynamoDB trigger settings isn't a supported option.

Given these constraints, my primary question is: how can I effectively handle transient errors when using a Lambda Trigger? It's essential for me to maintain the chronological order of the changes in the DynamoDB stream to ensure accurate replication to OpenSearch. Any insights into a suitable strategy for accomplishing this would be greatly appreciated.

Didier_Durand · Answer

Hi,

You can envision Amazon OpenSearch Ingestion for your use case: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ingestion.html

Your Lambda trigger writes to an S3 bucket and from there use the Processor mechanism of Ingestion to push data in right order (preserving initial creation order in your case) to OpenSearch :

```
Processors
Intermediate processing units that can filter, transform, and enrich records into 
a desired format before publishing them to the sink. The processor is an optional 
component of a pipeline. If you don't define a processor, records are published in 
the format defined in the source. You can have more than one processor. A pipeline 
runs processors in the order that you define them.
```

Re. cost-effectiveness look at section re. pipeline capacity : you can adapt min OCU ingestion capacity to have very minimal costs when you don't use the pipeline.

Best,

Didiuer

Suggest a failover mechanism when replicating from DynamoDb Table to OpenSearch

Relevant content