Amazon RDS with Amazon Opensearch Service

Hello.

It is also possible to use AWS DMS.
AWS DMS allows you to set up continuous replication from RDS to Opensearch Service.
https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html

If it's acceptable for the synchronization timing to be somewhat delayed, using methods involving Glue or Lambda could also be appropriate.

Hi,

I would use AWS DMS for such a task and use its features to first export to S3.

Then, I'd place a Lambda trigger on this bucket to detect each data addition by DMS. Adding this Lambda will give you more flexibility in the data processing that you may need to do before loading into S3. DMS to OpenSearch doesn't offer such flexibility.

Best,

Didier

AWS DMS can be used for this use case. AWS DMS can have RDS as source and OpenSearch as target. You can do FULL LOAD and CDC to OpenSearch directly. Ref

RDS -> DMS -> OpenSearch

If you want to apply any transformation to data before sending to OpenSearch like concatenating columns, removing PII info etc, then you can use lambda in between.

RDS -> DMS -> S3 -> Lambda -> OpenSearch

Note: Here you need to write the program to read the data apply the transformation and load to OpenSearch in Lambda. Instead of lambda you can use Amazon OpenSearch Ingestion. This is a fully managed, serverless data collector that delivers real-time log, metric, and trace data to Amazon OpenSearch Service domains and OpenSearch Serverless collections. OpenSearch ingestion filter, transform, and enrich records into a desired format before publishing them to the OpenSearch. OpenSearch Ingestion has the following main benefits:

• Eliminates the need for you to manually manage a self-provisioned pipeline.
• Automatically scales your pipelines based on capacity limits that you define.
• Keeps your pipeline up to date with security and bug patches.
• Provides the option to connect pipelines to your virtual private cloud (VPC) for an added layer of security.
• Allows you to stop and start pipelines in order to control costs.
• Provides pipeline configuration blueprints for popular use cases to help you get up and running faster.
• Allows you to interact programmatically with your pipelines through the various AWS SDKs and the OpenSearch Ingestion API.
• Supports performance monitoring in Amazon CloudWatch and error logging in CloudWatch Logs.

With OpenSearch ingestion service pipeline will look like below

RDS -> DMS -> S3 -> OpenSearch Ingestion Service -> OpenSearch