In the previous article we established why distributed systems exist — fault tolerance, scalability, and latency eventually force you off a single machine. Once you accept that, a new question immediately follows: if your data lives on multiple machines, how do you keep them in sync?
This is replication. And it is one of those topics that sounds simple until you look closely.
Why Replicate at All
Replication means keeping a copy of the same data on multiple nodes. The reasons to do it map directly to the three pressures from the scaling article.
Fault tolerance. If one node fails, the others continue serving requests. No single failure takes down the entire system. This is the primary reason most teams add replication — not scale, but survival.
Read throughput. If five nodes hold the same data, read requests can be spread across all five. Each node handles a fifth of the read load. For read-heavy workloads this is a significant win.
Latency. You can place replicas geographically close to your users. A replica in Singapore serves Southeast Asian users faster than a primary in Frankfurt ever could, regardless of how fast the network is.
These benefits are real. But replication introduces a problem that has no clean solution: what happens when the data changes?
The Fundamental Problem: Replication Lag
When a write arrives at one node, that node needs to communicate the change to every other node holding a copy. This takes time. During that time, different nodes hold different versions of the data. This gap is called replication lag.
How you handle replication lag is the central engineering decision in any replicated system. There are two approaches.
Synchronous Replication
The primary node waits for confirmation from replicas before telling the client the write succeeded. The client gets a success response only when the data is safely written to multiple nodes.
Client → Primary: write value="Delhi"
Primary → Replica 1: replicate
Primary ← Replica 1: confirmed
Primary → Client: success ✅
The advantage: when the client gets a success response, the data is guaranteed to be on multiple nodes. If the primary dies immediately after, a replica can take over without losing the write.
The disadvantage: every write is slower. The primary has to wait for the network round trip to each synchronous replica. If a replica is slow or unreachable, writes stall. In practice, having all replicas synchronous is usually impractical — one slow replica blocks every write in the system.
Asynchronous Replication
The primary tells the client success as soon as it has written locally, then replicates to followers in the background.
Client → Primary: write value="Delhi"
Primary → Client: success ✅ (immediately)
Primary → Replica 1: replicate (background, may take milliseconds or seconds)
The advantage: writes are fast. The primary doesn’t wait for anyone.
The disadvantage: if the primary fails before replication completes, the write is lost even though the client received a success confirmation. The replica that takes over as the new primary won’t have the write.
The Pragmatic Middle Ground
Most databases default to asynchronous replication with one synchronous replica — a compromise called semi-synchronous. At least one replica is guaranteed to have the latest write, but the rest catch up asynchronously. You get durability without making every write wait for all replicas.
What Replication Lag Actually Looks Like
In a healthy system with a fast network, replication lag might be a few milliseconds — barely perceptible. But in practice, lag grows under load, across geographic distances, or when a replica is catching up after being offline.
When replication lag is significant, three specific problems appear. Each one is a real bug that real users experience.
Read-Your-Own-Writes
You submit a form on a web application. The page refreshes. Your change is gone.
This is the most common replication-related bug in production systems. Here is exactly what happens:
- You write to the primary node
- The primary acknowledges success
- Your next request is routed to a replica (for load balancing)
- The replica hasn’t received the write yet — replication lag
- The replica returns the old value
- Your update appears to have disappeared
From your perspective, you wrote something and it vanished. From the system’s perspective, everything worked correctly — the write went to the primary, replication is in progress, the replica is returning its current state. No errors were thrown.
The fix is read-your-own-writes consistency — a guarantee that after you write something, you will always see that write in subsequent reads. The simplest implementation: route reads for recently-modified data to the primary. If you just updated your profile, read your profile from the primary for the next few seconds, not from a replica.
It sounds simple but gets complicated quickly. If a user edits from multiple devices, “recently modified by this user” becomes hard to track. If the primary is down, you can’t read from it. The details matter.
Monotonic Reads
You refresh a page and see a comment someone just posted. You refresh again and the comment is gone. You refresh a third time and it’s back.
This happens when consecutive reads land on different replicas with different amounts of lag. Replica A has the comment (low lag), Replica B doesn’t yet (high lag). Your first read hits A, your second hits B, your third hits A again.
The data appears to go backwards in time. Not because anything was deleted — just because different replicas are at different points in the replication stream.
Monotonic reads is the guarantee that a user will never see time go backwards. If you’ve seen a value at time T, you will never subsequently see a value from before T. The implementation typically involves routing each user’s reads to the same replica consistently, so they always see a monotonically advancing view of the data.
Consistent Prefix Reads
A conversation between two people:
Alice: “How far in the future can you predict the weather?” Bob: “About ten days.”
If you read this conversation from a replica that received Bob’s reply before Alice’s question (because they were written to different partitions that replicated at different speeds), you see:
Bob: “About ten days.” Alice: “How far in the future can you predict the weather?”
The answer appears before the question. The conversation makes no sense.
This problem occurs specifically when writes to different partitions replicate at different rates. Consistent prefix reads is the guarantee that if a sequence of writes happens in a certain order, anyone reading those writes will see them in the same order.
What “Eventual Consistency” Actually Means
These three problems — read-your-own-writes, monotonic reads, consistent prefix reads — all occur when a system is described as “eventually consistent.”
Eventual consistency is often explained as “the data will be consistent eventually.” That description is technically true but practically useless. What it actually means is: if you stop writing to the system and wait long enough, all replicas will converge to the same value. It says nothing about how long that takes, or what you will see in the meantime.
The three bugs above are not edge cases. They are the everyday reality of eventual consistency under normal operation. They appear on every page load when replication lag exists, which is most of the time.
This is not an argument against eventual consistency — it is the right tradeoff for many systems. A social media timeline that shows a post a few seconds late is not broken. An inbox counter that shows 3 unread instead of 4 for a moment is not broken. The system is working as designed.
But it does mean that choosing eventual consistency is a real decision with real consequences. The question is not “is eventual consistency good or bad?” It is “can my application tolerate users occasionally seeing stale data, and have I handled the specific failure modes it produces?”
The Tradeoff in One Sentence
Stronger consistency guarantees require more coordination between nodes, which means higher latency and lower availability. Weaker consistency guarantees require less coordination, which means lower latency and higher availability — but users sometimes see stale data.
Every replicated system sits somewhere on this spectrum. The right position depends on what your application can tolerate, not on what sounds better in an architecture meeting.
What Comes Next
This article covered the foundation — why replication exists, the core synchronous versus asynchronous tradeoff, and the three user-visible bugs that replication lag produces.
But it hasn’t addressed a more fundamental question: which node accepts writes? Can any node accept a write, or is there a designated primary? What happens when the primary fails?
Those questions are about replication architecture — single-leader, multi-leader, and leaderless — and they’re covered in the next article.