At a Glance
- CAP: during a network partition, you must pick Consistency or Availability — never both.
- P is not optional in a real distributed system; the only real choice is CP vs AP.
- CP (ZooKeeper, etcd) refuses to answer rather than lie. AP (Cassandra, DynamoDB) answers, possibly with stale data.
- Real systems sit on a spectrum — Cassandra's consistency level is tunable per query.
- PACELC adds the everyday trade-off: even without partitions, you trade Latency vs Consistency.
- Google Spanner bends the rules with atomic clocks — hardware can shift what's possible.
The Three Properties
In 2000, Eric Brewer conjectured (and Gilbert & Lynch later proved) that a distributed data store can guarantee at most two of these three properties at the same time:
C — Consistency
Every read receives the most recent write (or an error). All nodes agree on one truth, as if there were a single copy of the data.
A — Availability
Every request to a non-failed node receives a non-error response — without any guarantee that it contains the latest write.
P — Partition Tolerance
The system keeps operating even when network messages between nodes are lost or delayed — i.e., when the cluster splits into islands.
Pick an edge that touches P — because in a real network, P is not optional.
Why You Can Only Pick Two
Imagine two database nodes, Node A in Virginia and Node B in Frankfurt, replicating to each other. A backhoe cuts the cable between them — a network partition. Now a client writes balance = $500 to Node A, and another client reads the balance from Node B.
Node B has not received the update, and it can't reach Node A to check. It has exactly two choices:
"I can't guarantee I'm up to date, so I'll return an error or block until the partition heals." Reads are always correct, but the system is partially unavailable.
"Here's the last value I saw." Every request gets a response, but two clients can briefly see two different truths.
There is no third option. Answering correctly requires communication; the partition forbids communication. That's the whole theorem.
Switch failures, garbage-collection pauses, overloaded NICs, fat-fingered firewall rules — anything that delays messages long enough behaves like a partition. In any system spanning more than one machine, you must tolerate P. So the real-world choice is always CP vs AP, never "CA."
Why Partition Tolerance Is Non-Negotiable
Newcomers sometimes ask, "Can't I just pick C and A and skip P?" No — and understanding why is the whole point of CAP. Network partitions happen in real distributed systems:
Bad cables & switches
Hardware fails constantly at scale. A top-of-rack switch dying partitions every server in the rack.
BGP routing issues
A bad route announcement can make two healthy datacenters mutually unreachable for minutes.
Datacenter fires & floods
OVH's 2021 fire took an entire datacenter offline. Region-level partitions are rare but real.
GC pauses & NAT timeouts
A 30-second garbage-collection pause is indistinguishable from a dead node. Slow is the new down.
When a partition happens, nodes can't communicate. You have exactly two choices:
- Stop serving (sacrifice A) → consistent but unavailable.
- Keep serving with potentially stale or diverged data (sacrifice C) → available but inconsistent.
You cannot sacrifice P: that would mean "assume no partitions ever happen" — which is only true on a single machine, and a single machine isn't a distributed system. So the real question is never "do we tolerate partitions?" It's: when a partition DOES occur, which side do you sacrifice?
N3 and N4 can't see the latest write. Their only options: answer anyway (AP) or refuse (CP).
CP Systems: Correct, Sometimes Closed
CP systems prefer to return errors (or wait) rather than serve stale data. They typically use quorums or a single leader: if a node can't confirm it has the latest state, it steps aside.
- ZooKeeper / etcd — coordination services; a wrong answer about "who holds the lock" is catastrophic, so they go read-only or unavailable in a minority partition.
- HBase — each region has one serving node; if it's unreachable, that data is briefly unavailable rather than inconsistent.
- MongoDB (with majority read/write concerns) — a primary that can't reach a majority steps down and stops accepting writes.
- Redis (single node) — trivially consistent; if the node is down, you simply get nothing.
Choose CP when wrong answers are worse than no answers: bank balances, inventory counts, distributed locks, leader election.
AP Systems: Always On, Eventually Right
AP systems keep accepting reads and writes on both sides of a partition, then reconcile afterwards using version vectors, last-writer-wins, or application-level merge logic.
- Cassandra — any replica accepts writes (tunable); divergent replicas heal via read repair and anti-entropy.
- DynamoDB — born from Amazon's insight that an unavailable shopping cart costs real money; stale carts can be merged later.
- CouchDB — built for multi-master, even offline-first replication with explicit conflict revisions.
- DNS — the world's largest AP system: resolvers serve cached (possibly stale) answers for hours, and nobody waits for global agreement.
Choose AP when availability is the product: social feeds, shopping carts, likes and view counters, presence indicators.
| CP (consistency first) | AP (availability first) | |
|---|---|---|
| During a partition | Errors / blocked requests on minority side | All nodes keep answering |
| Risk | Downtime | Stale reads, write conflicts |
| Conflict handling | Prevented up front (quorum/leader) | Resolved after the fact (merge, LWW) |
| Examples | ZooKeeper, etcd, HBase, MongoDB | Cassandra, DynamoDB, CouchDB, DNS |
| Great for | Money, locks, inventory | Feeds, carts, counters, sessions |
The Spectrum, Not a Triangle
Martin Kleppmann's famous critique — "Please stop calling databases CP or AP" — points out that the triangle picture misleads in two ways. First, "consistency" means something different in CAP than in databases: CAP's C = linearizability, a very strong property. Most systems labeled "CP" don't actually guarantee CAP's C. Second, real systems aren't pinned to a vertex — they live on a tunable spectrum.
Cassandra makes this concrete: you choose a consistency level per query:
| Consistency level | Behavior | Where it sits |
|---|---|---|
ONE | Ack from a single replica — fast, possibly stale | AP-like (available, potentially stale) |
QUORUM | Majority of replicas must agree | Balanced — strong-ish reads, survives a node loss |
ALL | Every replica must respond | CP-like (consistent, fails if any node is down) |
- "CP or AP?" is a simplification. The real question: what consistency level does this use case need?
- The same cluster can serve a strongly-consistent billing query and an eventually-consistent feed query side by side.
- In interviews, naming the knob (ONE / QUORUM / ALL) beats naming the vertex.
Spanner — When You Have Atomic Clocks
Google Spanner is the system people point at when they say "but CAP says you can't do that!" It's an externally consistent, distributed SQL database that spans continents — and it bends the trade-off with hardware.
Every datacenter has GPS receivers + atomic clocks. Instead of "the time is X," TrueTime answers "the time is within [X−ε, X+ε]" with ε ≈ ±7 ms of global uncertainty.
Spanner holds a transaction open until the uncertainty window has fully passed before making it visible. Result: timestamps are globally ordered → linearizable transactions across continents.
Brewer himself describes Spanner as effectively CA: partition tolerance is engineered to be so good (redundant fiber, multiple datacenters per region) that P is rarely invoked. Technically it's CP — when a real partition hits, it chooses consistency.
CAP describes trade-offs given your infrastructure — and hardware changes can shift what's possible in the CAP space. Atomic clocks didn't repeal the theorem; they made the painful branch of it astronomically rare.
PACELC: The Grown-Up Version
CAP only talks about behavior during a partition — but partitions are rare. What about the other 99.99% of the time? Daniel Abadi's PACELC fills the gap:
Partition? → choose A or C · Else → choose Latency or Consistency
Even on a healthy network, strong consistency requires waiting for replicas to confirm — which costs latency. Skip the wait and you're fast but possibly stale. Every distributed database lives somewhere on this spectrum:
- PA/EL (favor availability and latency): Cassandra, DynamoDB — fast and forgiving.
- PC/EC (favor consistency always): HBase, ZooKeeper, Google Spanner — correct and patient.
- PA/EC and hybrids: MongoDB and others let you tune per-query.
Why the latency trade-off exists even on a healthy network
Synchronous replication
Strongly consistent — the write isn't acknowledged until all (or a quorum of) replicas confirm. But you must wait for the slowest replica's ack, which adds latency to every single write.
Asynchronous replication
Low latency — ack immediately, replicate in the background. But a replica might be behind, so reads from it can be inconsistent. You traded C for L without any partition in sight.
| System | During partition | Normal operation | Character |
|---|---|---|---|
| DynamoDB | A — keep serving | L — prioritize latency | Fast, eventually consistent |
| ZooKeeper | C — refuse if no quorum | C — wait for consensus | Slow but always consistent |
| Cassandra | A — keep serving | L — prioritize latency | Tunable per query |
| Spanner | C — choose consistency | C — commit-wait | Google's magic clocks |
In an interview, don't just say "CAP theorem!" and stop. Say which letter you're sacrificing, where, and why: "The payment ledger is CP via quorum writes; the activity feed is AP with eventual consistency, because a stale like-count never hurt anyone."
Quick Check
During a network partition, a Cassandra replica happily serves a read that might be slightly out of date. Which trade-off is it making?
Who Chose What — Real Systems, Real Stakes
ZooKeeper (CP)
Used by Kafka for leader election and HDFS for the name node. Chooses consistency — if a quorum is unavailable, ZK stops answering rather than risk divergence. A wrong answer about "who is the leader" would be catastrophic.
Cassandra (AP)
Used by Netflix, Instagram, and Discord. Chooses availability — 3 replicas, configurable quorum, tunable consistency per query. Built to keep accepting writes through node failures and partitions.
DynamoDB (AP)
Powers Amazon's own cart and order systems. Single-digit-millisecond latency at any scale; born from the insight that an unavailable shopping cart costs real money.
etcd (CP)
The Kubernetes control plane. Leader election and configuration storage — the cluster's source of truth must be consistent, even at the cost of brief unavailability.
Interview Q&A
Trick question — it depends on configuration. With the default write concern {w:1} it's AP-ish (available, with risk of data loss on failover). With {w:"majority", readConcern:"majority"} it approaches CP behavior. The "MongoDB is CP" answer from RDBMS land is a simplification. This is the key insight: most real systems are configurable on the spectrum.
Split-brain: both sides of the partition think they're the primary and accept writes. When the partition heals, you have conflicting state. Solutions: vector clocks to detect conflicts, CRDTs for auto-merging, last-writer-wins (lossy), or application-level conflict resolution (Dropbox shows a "conflicted copy"). Dynamo (Amazon) uses all of these — the 2007 Dynamo paper is required reading.
Shopping cart: it's fine if two devices show slightly different items temporarily — they converge. DNS: cached stale records are acceptable for the TTL duration. Social media like counts: showing 104,829 vs 104,830 likes doesn't matter. Counter-example: a bank balance — you never want to read a stale balance and allow an overdraft.
Anti-Patterns to Avoid
"We chose CP so we have no availability issues"
Being CP means you sacrifice availability during partitions. ZooKeeper can be unavailable if quorum is lost — that's the deal you signed.
Assuming AP means "all writes always succeed"
AP systems still reject writes when they can't meet their own guarantees. Cassandra with QUORUM won't write if fewer than ⌊N/2⌋+1 nodes are up.
Applying CAP to single-node databases
CAP is about distributed systems. SQLite is neither CP nor AP — it's just a local file. No partition, no theorem.
Further Reading
Interactive: CAP Theorem Simulator
Trigger a network partition and choose whether to sacrifice Consistency or Availability. Watch how each choice changes the response your clients receive.