Best practices on dealing with eventual consistency with read replica


Hi all, My scenario is a database cluster with writer and reader instances. When the application persists new data, there's a small lag that's needed for the reader instance to update. The lag is being observed when attempting to read the data, causing the application to not be able to read the data previously persisted.

What are the best practices recommended to deal with the replica lag at application level?

1 Answer

There are a few things you can keep in mind in dealing with read-after-write staleness. First, do you really need to immediately read after the write? If you just successfully inserted a row, you know what it should look like, right? Second, if you made some kind of incremental change to a column's values, you can probably find some way to have the database return that row to you as part of the response to the write - for SQL, take a look at the RETURNING clause - for DynamoDB take a look at the ReturnValues parameter on the UpdateItem API. Finally, if you are making some kind of more complex query after the write (involving multiple rows etc) and you must see the result of that write within your query result, you might just need to direct the query to the leader node (just as you did for the write).

BTW - it's important to realize that when you read after the write, there's no guarantee that you'll see your data just as it was when your write was applied - it's always possible that some other process came along and changed things in the meantime!

answered 2 months ago
profile picture
reviewed 21 days ago
  • Hi, thanks for your answer. What do you recommend when the application that reads is not the same that writes? In my case, the writer application sent an event to a message queue after the write, and another application reacted to the event and attempted to read the data. What I observe is sometimes the event is faster than the database replication.

  • Got it - so the reader is quite independent from the writer. I think there are probably a couple of different approaches for this. First would be to direct the read from the event driven process to the leader replica - the writer endpoint. Another would be to stay with making reads from the read replicas, but introduce some delay in the event processing - perhaps with some retries to handle scenarios when the (I would suggest using an exponential back-off on these retries).

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions