Issues with ongoing loads/replication/delta loads using DMS

0

Need your help as this falls outside my expertise, here is my customer question:

Currently we are using AWS Database Migration service (DMS) to do one time loads and ongoing loads into red-shift from different Sources.

DMS is great when we doing One time loads. However there are some issues with ongoing loads/ replication/ delta loads.

Sometimes counts are not matching with the error messages log sequence file is missing, even though that particular log file exists on the server.

Solution from support is restart the task. Since we are using this tool in production for daily feeds, restarting the task whenever something issues arises is not a good idea.

Coming to the SQL server, again one time loads are fine with minor problems but live replication needs too many functions on the SQL server like CDC, Replication etc.

Since Our SQL servers are on AlwaysOn tech, adding replication to the server will add load, increases latency and requires more man power to fix the replication issues. Always on is more advanced feature than replication with less maintenance than replication.

Are there any alternative tools to avoid all these problems? something like AWS Glue, AWS data pipeline, any other third party tools.

1 Answer
0
Accepted Answer

Almost all the vendors supporting ongoing replication are based on transaction log- a feature that Oracle, SQL Server and MySQL have. This applies to AWS DMS as well as 3rd party tools such as Attunity.

If real time replication is not a requirement then one option is to use a JDBC connector program that can be scheduled to run. The rational of such program is to track changes according to an incremental column in the table- either timestamp or unique sequential value. This is similar to https://redshiftsupport.matillion.com/customer/portal/articles/2955330.

AWS
answered 5 years ago
profile picture
EXPERT
reviewed 23 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions