CodeOath
← All posts
System Design52 min total · 14 parts

System Design Fundamentals for Interviews: Scalability, Trade-offs, and the Framework Interviewers Actually Grade

Part 7 of 14 · ~4 min

Consistency Models and the CAP Theorem

Every scaling technique in the last chapter bought capacity by adding copies of data — replicas, shards, caches — and every one of those copies raises the same question: when the copies disagree, even briefly, what does the system promise you?

Strong vs. Eventual Consistency

Strong consistency means that once a write completes, every subsequent read — from any node, immediately — sees that write. There's no window where a client can observe stale data; a real database transaction on a single leader gives you exactly this.

Eventual consistency relaxes that guarantee: after a write, the system promises that if no new writes happen, all replicas will eventually converge on the same value — but makes no promise about how long "eventually" takes, and a read that lands on a replica during that window can return stale data. A follower that hasn't finished replicating the leader's latest write, from the previous chapter, is exactly this in practice.

The trade is latency and availability against a real, if brief, staleness window. Strong consistency generally requires coordinating with multiple nodes (or a single authoritative one) before a write — or a read — can be considered done, which costs time and can't be satisfied at all if the required nodes aren't reachable. Eventual consistency lets any reachable replica answer immediately, at the cost of that replica occasionally being behind.

What CAP Actually Claims

The CAP theorem states that a distributed data store can provide at most two of the following three guarantees simultaneously, in the presence of a network partition:

  • Consistency — every read receives the most recent write (or an error) — this is the strong consistency defined above, specifically in the CAP context.
  • Availability — every request to a non-failed node receives a (non-error) response — it doesn't have to be the most recent value, just some valid response.
  • Partition tolerance — the system continues operating despite network partitions (messages between nodes being dropped or delayed).

The most commonly misunderstood part of CAP is treating it as "pick any two of C, A, P as a permanent, general design choice." In practice, network partitions are going to happen in any real distributed system — cables get cut, switches fail, packets get dropped — so partition tolerance isn't really an option you get to decline; it's a fact about operating over an unreliable network. The actual choice CAP describes only kicks in during a partition: when part of the system genuinely cannot reach another part, do you return a possibly-stale-but-available answer anyway (choosing AP), or do you refuse to answer until you can guarantee correctness (choosing CP)? Outside of an active partition, a well-designed system can very reasonably offer both consistency and availability at once — CAP has nothing to say about the normal, non-partitioned case at all, which is most of the time.

CP (consistent, partition-tolerant)AP (available, partition-tolerant)
Behavior during a partitionRejects requests it can't guarantee are correctKeeps answering, possibly with stale data
Good fitFinancial balances, inventory counts — being wrong is worse than being briefly unavailableSocial media feeds, product catalogs, "likes" counts — a stale view is fine, an error page isn't
Example systems (by configuration)Traditional relational databases with synchronous replication, ZooKeeper, etcdDynamoDB (default config), Cassandra (default config), most CDN-cached content

PACELC: The More Practically Useful Extension

CAP only describes behavior during a partition, which is a fairly rare event compared to how often a distributed system is actually running normally. PACELC extends the idea to cover the much more common case: if there's a Partition (P), choose between Availability and Consistency (A/C) — Else (E, i.e., no partition), choose between Latency and Consistency (L/C).

That second half is the genuinely useful addition: even with no partition at all, a system that wants strong consistency across replicas has to pay a coordination cost — waiting for enough replicas to acknowledge a write, for instance — that a system willing to accept eventual consistency simply doesn't pay. PACELC is why a system can honestly describe itself as, say, "PA/EL" — available over consistent during a partition, and low-latency over strongly-consistent the rest of the time — which is a far more complete and decision-relevant description than "AP" alone, since it also tells you what the system is optimizing for on an ordinary Tuesday with no partition in sight.