At a Glance
- Consistency spectrum: weak → eventual → strong. Pick the cheapest each feature can tolerate.
- Quorum math: with N replicas, W + R > N guarantees reads see the latest write.
- Vector clocks track causality without a global clock; CRDTs merge conflicts automatically.
- Series availability multiplies down; parallel redundancy multiplies failure away.
- RPO = how much data you can lose; RTO = how long you can be down.
- Retry with exponential backoff + jitter, and test failure with chaos engineering.
Consistency Patterns: How Stale Can a Read Be?
When data lives on multiple nodes, "what does a read return right after a write?" becomes a spectrum. Three useful points on it:
1. Weak consistency
After a write, reads may or may not ever see it. No promises. This sounds terrible until you realize it's perfect for real-time data where late information is worthless anyway: if you drop a few frames of a video call or a position update in a game, you want the newest data next, not a replay of the old.
Used by: VoIP, video chat, multiplayer game state, memcached under contention.
2. Eventual consistency
After a write, reads will see it eventually — typically within milliseconds to seconds — once replication catches up. The system converges to one truth, just not instantly.
Used by: DNS propagation, email delivery, S3 (historically), Amazon shopping carts, social media timelines.
N1 accepts the write instantly; N2 and N3 converge moments later. Readers in between may see history.
3. Strong consistency
After a write completes, every read sees it, everywhere. Achieved by synchronous replication, quorums, or routing all reads through the leader. The price is latency (you wait for replicas) and reduced availability (no majority → no service).
Used by: bank ledgers, payment systems, seat/inventory reservation, anything in an RDBMS transaction, Google Spanner.
| Pattern | Guarantee | Cost | Great for |
|---|---|---|---|
| Weak | None — best effort | — | Calls, gaming, live telemetry |
| Eventual | Converges after replication lag | Brief staleness | Feeds, carts, DNS, email |
| Strong | Reads always see latest write | Latency + availability | Money, inventory, transactions |
One product can use all three. An e-commerce site reads product pages eventually-consistently, tracks your cart eventually, and charges your card strongly. Pick the cheapest consistency each feature can tolerate.
Quorum: The Math Behind Distributed Consensus
How do replicated systems offer strong consistency without routing everything through one leader? Quorums. With N replicas, require W nodes to acknowledge each write and consult R nodes on each read. The magic inequality:
W + R > N # strong consistency: read & write sets MUST overlap
# Example: N=3, W=2, R=2 → W+R = 4 > 3
# Any 2 written nodes overlap any 2 read nodes
# → at least one node in every read has the latest writeWhatever 2 nodes the write touches and whatever 2 nodes the read consults, at least one node is in both.
Tuning W and R lets you slide along the consistency/latency spectrum without changing the architecture:
| Configuration (N=3) | Writes | Reads | Personality |
|---|---|---|---|
| W=2, R=2 | Medium | Medium | Balanced strong consistency |
| W=1, R=3 | Fast | Slow | Write-heavy workloads, consistent reads |
| W=3, R=1 | Slow | Fast | Read-heavy — like a CDN's profile |
| W=1, R=1 | Fast | Fast | W+R ≤ N → eventual consistency only |
Cassandra exposes exactly this: you choose a consistency level (ONE, QUORUM, ALL) per query, which is just choosing W and R on the fly.
Vector Clocks — Tracking Causality
Distributed systems have no global clock — every machine's clock drifts, and NTP can only do so much. So how do you know which of two writes happened first, or whether they happened concurrently? Logical clocks.
The simplest logical clock is a counter each node increments on every event. A vector clock extends this: each node tracks a vector of counters, one slot per node — [A:2, B:1, C:0] means "I've seen 2 events from A, 1 from B, none from C."
# Vector clock rules (node A's perspective)
on local event:
clock["A"] += 1
on send(message):
clock["A"] += 1
message.clock = clock # piggyback the vector
on receive(message):
for node in all_nodes: # elementwise max
clock[node] = max(clock[node], message.clock[node])
clock["A"] += 1
# Comparing two versions:
# [A:2,B:1] vs [A:3,B:1] → second DOMINATES → it's newer
# [A:2,B:1] vs [A:1,B:2] → NEITHER dominates → concurrent → CONFLICT!- If one vector is ≥ the other in every slot, it causally descends from it — safe to overwrite.
- If neither dominates, the writes were concurrent — the system has detected a genuine conflict instead of silently losing data.
- Used by DynamoDB for version tracking, plus Riak and CouchDB.
CRDTs — Auto-Merging Conflicts
Vector clocks detect conflicts; someone still has to resolve them. Conflict-free Replicated Data Types go further: they're data structures mathematically designed so that any two divergent copies always merge to the same correct result — no coordination, no conflict resolution code.
G-Counter
Grow-only counter. Each node increments only its own slot; merge = elementwise max; total = sum of slots. No conflict is even possible.
PN-Counter
Two G-Counters: one for increments, one for decrements. Value = P − N. Perfect for like counts and subscriber counts.
LWW-Register
Last-Writer-Wins register with timestamps. Simplest CRDT — but if clocks skew, the "loser" write silently disappears. Use with care.
Figma uses CRDT-style techniques for collaborative editing — multiple designers editing the same layer simultaneously, merging without locks. Apple Notes sync, Riak, and Azure Cosmos DB all use CRDT-based approaches. When you need multi-master writes with zero coordination, CRDTs are the principled answer.
Availability Patterns: Surviving Failure
Availability comes from one idea — redundancy — applied two ways: failover (a spare takes over) and replication (multiple copies serve at once).
Active-passive failover
One node serves traffic (active); a second waits in the wings (passive). They exchange heartbeats — periodic "still alive?" pings. If heartbeats stop, the passive node takes over the IP address or DNS name and starts serving.
- Hot standby: fully running and synced — takeover in seconds.
- Warm standby: running but lightly synced — takeover in minutes.
- Cold standby: powered off until needed — cheapest, slowest to recover.
Drawbacks: you pay for hardware that's idle most of its life, and there's a blip of downtime during failover. If the standby's data lags, recent writes can be lost.
Active-active failover
Both nodes serve traffic simultaneously, with a load balancer (or geo-DNS) spreading requests between them. Lose one and the other simply absorbs the load. No idle hardware — but now both nodes must handle writes or share state, which drags in the replication problems below.
Master-slave (primary-replica) replication
One primary takes all writes and streams its change log to read-only replicas. Reads scale beautifully — just add replicas. The catch is replication lag: read a replica right after writing the primary and you might not see your own write. If the primary dies, a replica must be promoted (manually or via election), and unreplicated writes can vanish.
Master-master (multi-primary) replication
Every node accepts writes and syncs with the others. Survives any single node loss with no promotion dance — but two nodes can accept conflicting writes to the same row at the same moment. Now you need conflict resolution: last-writer-wins (data loss risk), CRDTs, or app-level merging. Many teams sidestep conflicts entirely by partitioning writes (odd IDs to node 1, even to node 2).
| Pattern | Writes | Failure behavior | Main pain |
|---|---|---|---|
| Active-passive | One node | Standby promoted on heartbeat loss | Idle spare, failover blip |
| Active-active | Both nodes | Survivor absorbs traffic | State sync complexity |
| Master-slave | Primary only | Replica promotion needed | Replication lag, lost tail writes |
| Master-master | All nodes | Seamless | Write conflicts |
Active-Passive vs Active-Active Deep Dive
The failover patterns above deserve a closer look, because the gap between "cold standby" and "active-active" is the gap between minutes of downtime and none:
The passive server only receives heartbeats; data syncs periodically. Failover time: seconds to minutes. RPO depends on sync frequency — you lose whatever wasn't synced. Simplest and cheapest, but you pay for capacity that sits idle.
The passive server mirrors the primary in real time and can take over in seconds. Near-zero RPO. More expensive — you're running two full systems but serving from one.
Both nodes handle traffic; the load balancer splits requests. If one dies, the other takes over instantly (near-zero RTO) — it was already serving. No wasted capacity.
- Active-active's challenge is state sync: writes to node A must reach node B, or requests will diverge depending on where the LB sends them.
- Stateless services do active-active trivially — your web tier already does this behind a load balancer.
- Stateful services (a primary database) can't do naive active-active without conflict resolution — that's master-master replication with all its pain.
RTO and RPO — The Disaster Recovery Axes
Two numbers define every disaster recovery plan. Agree on them with the business before the disaster:
RPO — Recovery Point Objective
How much data loss is acceptable? "If disaster strikes at 3pm, how old can our last good copy be?" RPO=0: synchronous replication, zero loss, expensive. RPO=1hr: hourly snapshots, cheap, lose up to an hour of work.
RTO — Recovery Time Objective
How long can the system be down during recovery? RTO≈0: active-active with instant failover. RTO=15min: warm standby plus some human involvement. RTO=4hr: restore from backup, significant downtime.
Closer to the origin = less loss, less downtime, more money. Pick the point your business actually needs.
Availability Math: Series vs Parallel
This is the most useful arithmetic in this course. When a request must pass through services in sequence, availabilities multiply — and multiply means shrink:
# In series (request needs BOTH): availability drops
LB(99.9%) → App(99.9%) = 0.999 × 0.999 ≈ 99.8%
add DB(99.9%) = 0.999³ ≈ 99.7%
# In parallel (request needs EITHER): availability soars
two 99.9% replicas = 1 − (0.001 × 0.001) = 99.9999%Every dependency you chain in series eats your nines; every redundant replica you add in parallel multiplies failure probabilities into oblivion. This single fact explains why architects obsess over removing synchronous dependencies and adding replicas.
Parallel math only works if failures are independent. Two replicas in the same rack share a switch; two AZs share a region; everything shares your deploy pipeline. Correlated failures are why "five nines on paper" systems still make headlines.
SLAs, SLOs, and SLIs
Three acronyms that turn "the site should be up" into engineering:
The measurement itself. "p99 latency over 5 minutes" or "fraction of 2xx responses." If you can't graph it, it's not an SLI.
Your internal target for the SLI: "99.95% of requests succeed each month." Miss it and pagers ring; the gap between 100% and the SLO is your error budget for risky deploys.
The contractual promise to customers, with penalties (usually service credits) for breaking it. Always looser than the SLO — you promise 99.9% while targeting 99.95%, so you breach your own alarm before you breach the contract.
Error budgets are sneaky-brilliant: if you're well within SLO, ship fast and take risks. If you've burned the budget, freeze features and fix reliability. It turns "devs vs ops" arguments into arithmetic.
Exponential Backoff with Jitter
When a dependency fails, clients retry. Done naively, retries make outages worse:
- Naive retry: fail → wait 1s → retry → fail → wait 1s → retry. Every client retries on the same schedule, so they all hammer the recovering service in synchronized waves — a thundering herd that knocks it right back down.
- Exponential backoff: wait 1s, 2s, 4s, 8s, 16s… (with a max cap). Pressure decays instead of staying constant.
- Jitter: add random noise — wait(1 + random(0,1)), wait(2 + random(0,2))… Clients desynchronize, spreading load evenly instead of arriving in spikes.
import random, time
def retry_with_backoff(operation, max_retries=5, base=1.0, cap=30.0):
for attempt in range(max_retries):
try:
return operation()
except TransientError:
if attempt == max_retries - 1:
raise # out of retries
backoff = min(cap, base * 2 ** attempt) # 1, 2, 4, 8, 16…
sleep_for = random.uniform(0, backoff) # "full jitter"
time.sleep(sleep_for) # clients desynchronizeThe AWS SDK does exponential backoff with jitter by default. Any distributed system that retries needs it — AWS's own benchmarks showed full jitter cuts total work and completion time dramatically versus plain backoff.
Chaos Engineering: Break It on Purpose
You can't claim your failover works if you've never watched it fire. Chaos engineering injects failures deliberately — in production — so you discover the gaps on a calm Tuesday instead of during the real outage.
Chaos Monkey (2010)
Netflix's original: randomly kills production servers during business hours. If an engineer's service can't survive a dead instance, they find out fast.
Chaos Gorilla
Bigger stick: kills entire availability zones to prove the system survives a datacenter-scale failure.
Latency Monkey
Injects artificial latency between services — because "slow" is harder to handle than "dead," and timeouts are where bugs hide.
FIT
Netflix Failure Injection Testing: controlled, scoped chaos in production — fail specific calls for specific users and watch the blast radius.
- Key insight: if you're not testing failure modes, you'll discover them at the worst possible time.
- Modern tooling: Gremlin, AWS Fault Injection Service, Chaos Mesh (Kubernetes).
- The principle: "Hope is not a strategy. Build systems that degrade gracefully."
Quick Check
A request flows through three services in sequence, each 99.9% available. Roughly how available is the whole path?
Interview Q&A
Three approaches: (1) Route all reads to the primary immediately after a write from that user — sticky routing for N seconds. (2) Track the replication position (LSN in Postgres) and route reads only to a replica that's caught up to that point. (3) Use synchronous replication for writes — the replica is guaranteed up-to-date before the write is acknowledged. Trade-offs: (1) adds load to the primary, (2) requires replica-lag tracking, (3) increases write latency.
HA means the system stays available despite failures — usually via redundancy and failover, which has some downtime window (seconds to minutes). Fault tolerance means the system continues operating without any service interruption even during a failure — much harder and more expensive (think NASA space shuttle computers with 5-way redundancy). In practice, most web systems aim for HA (99.99%), not true fault tolerance.
You're going from ~8.8 hrs/year of downtime to ~52 min: (1) Eliminate single points of failure — add redundancy to DB, LB, cache. (2) Multi-AZ deployment — spread across availability zones so a datacenter fire doesn't kill you. (3) Automated failover — manual intervention is too slow. (4) Health checks and circuit breakers — detect failures in seconds, not minutes. (5) Blue-green deployments — zero-downtime deploys. (6) Comprehensive monitoring with <1 min alert time. Each extra nine requires the previous nine plus more redundancy and better automation.
Anti-Patterns to Avoid
Single point of failure
One load balancer, one database, one DNS server. Always ask: "what happens if this dies?" — and keep asking until the answer is "nothing much."
Not testing failover
"We have a standby" means nothing if failover has never been rehearsed. Automating the test — like Netflix's Chaos Monkey — is table stakes.
Synchronous replication everywhere
Yes, it eliminates lag — but now every write waits for all replicas, and a slow replica slows your primary. Use async replication + read-your-own-writes techniques instead.
System Reliability
Reliability is the ability of a system to operate continuously without failure. It is distinct from simply "being up" — a reliable system must be correct, consistent, fault-tolerant, and available, even under adverse conditions.
What Reliability Actually Means
Four properties a reliable system must exhibit:
The system returns the right answer, not just any answer. An incorrect but fast response is worse than a slow correct one.
Users see a coherent view of data across requests and across nodes — not contradictory results.
Individual component failures do not cascade into system-wide outages. The system degrades gracefully, not catastrophically.
The fraction of time the system is reachable and serving requests within acceptable latency bounds.
MTBF and MTTR: Quantifying Reliability
The average duration a system operates before the next failure. A higher MTBF means failures happen less frequently. Improving MTBF means building more robust hardware, better software, and eliminating known failure modes.
The average time to detect a failure and restore service. A lower MTTR means you recover faster. Improving MTTR means better monitoring, automated failover, runbooks, and on-call discipline.
Availability ≈ MTBF / (MTBF + MTTR). To reach five nines, you need failures to be rare and recovery to be fast. Most teams underinvest in MTTR — cutting recovery time from 30 minutes to 2 minutes often delivers more availability than preventing failures entirely.
SLAs vs SLOs vs SLIs: The Conceptual Frame
The SLAs/SLOs/SLIs section earlier covered the math and error budgets. Here is the why behind the three-layer model:
- SLIs answer "what are we measuring?" — they are raw signals from your telemetry (request success rate, p99 latency, disk error rate).
- SLOs answer "what do we owe ourselves?" — internal targets that define when engineers must act. Breaching an SLO burns error budget; it is an internal alarm, not a public promise.
- SLAs answer "what do we owe customers?" — the external, contractual commitment, always set looser than the SLO so that your internal alarm fires before you breach the customer contract.
Without SLOs, engineers only learn about reliability problems when customers complain or contracts are breached. The SLO layer gives teams early warning and quantitative criteria for shipping vs. freezing feature work.
Availability vs Durability
These two properties are often conflated but address entirely different failure modes:
- Availability — the system is reachable and responsive. If your servers crash, availability drops. Example: your website returns 503.
- Durability — data written to the system is not lost. If your disk corrupts, durability drops. Example: AWS S3 boasts 11 nines of durability — data written is almost never lost even if nodes fail.
A system can be highly available but not durable (in-memory cache: fast but data evaporates on restart). It can be durable but not available (a cold backup tape: data is safe but inaccessible during an outage). Production systems need both.
Fault Tolerance vs High Availability
These terms are often used interchangeably but describe different engineering targets:
- High Availability (HA) minimizes downtime. When a component fails, the system detects it and fails over — there may be a brief interruption (seconds to minutes) while the secondary takes over. Most production web systems target HA.
- Fault Tolerance (FT) eliminates downtime from failures entirely. The system continues operating without any service interruption even during a failure. This requires much deeper redundancy — hardware-level mirroring, synchronous replication, lock-step execution — and is proportionally more expensive. Think NASA flight computers, not typical web services.
For most systems, aim for HA (four nines is 52 minutes of downtime per year). Reserve fault-tolerance engineering for the truly catastrophic failure modes — payment processing, medical devices, flight control. Over-engineering FT into a consumer app wastes money and adds complexity without proportional user benefit.
Reliability in Distributed and Cloud-Native Systems
Cloud infrastructure is inherently unreliable — VMs are preempted, networks partition, disks fail silently. The cloud-native reliability mindset flips the assumption: instead of preventing failures, you design for failure.
Multiple instances across zones and regions. No single node is indispensable.
Auto-scaling groups restart failed instances automatically. Health checks pull unhealthy nodes from rotation without human intervention.
Inject failures deliberately (see the Chaos Engineering section) so you discover gaps on a calm Tuesday, not during an incident.
Acknowledge network partitions are inevitable and choose your consistency/availability trade-off explicitly rather than by accident.
Reliability & Availability Interview Questions
Key questions on high availability, fault tolerance, reliability, and disaster recovery.
High Availability, Fault Tolerance & Failover Questions
High Availability (HA) refers to the ability of a system or component to be continuously operational and available with minimal downtime. The goal is to ensure that services are accessible and functional as much as possible, with systems designed to handle failures gracefully without significant disruption.
HA is crucial because modern systems — especially web-based applications — need to be reliable 24/7 to meet user expectations. Even short periods of downtime can lead to revenue loss, reputational damage, and customer dissatisfaction. Designing for HA means the system can withstand failures without causing prolonged service interruptions, ensuring a seamless user experience.
Active-Active: Multiple instances run in parallel sharing the workload. If one fails, others immediately take over with no service interruption. Provides better fault tolerance and load distribution but is more complex.
Active-Passive: One active node handles all traffic; passive nodes remain idle on standby until a failure occurs. Simpler to set up but may involve longer recovery times since the passive system must "take over" during failover.
When to choose: Active-Active when high fault tolerance, continuous operation, and load balancing are critical (mission-critical, high-traffic). Active-Passive when cost is a concern and the application can tolerate brief failover downtime.
Fault Tolerance is the ability of a system to continue operating even if one or more components fail. A fault-tolerant system maintains functionality and minimizes disruption through redundancy, failover mechanisms, and error detection.
Key difference from HA: High availability focuses on minimizing downtime — a server failure might trigger a failover with a brief interruption. Fault tolerance goes further by ensuring the system continues operating without any downtime at all during a failure. This requires much deeper investment — think redundant power supplies, memory mirroring, or NASA-style lock-step execution.
In practice: most web systems aim for HA (99.99%). True fault tolerance is reserved for medical devices, flight control, and financial core systems.
Graceful degradation means designing a system so that it continues to function with limited features or reduced performance when a failure occurs — rather than failing completely.
Examples:
- A shopping website with a payment outage still lets users browse and add items to cart rather than showing a complete "site down" error.
- A video streaming service degrades to lower resolution when bandwidth is constrained, so users can still watch.
This approach maintains user trust by avoiding complete failure and helping preserve some functionality during outages.
Load balancers distribute incoming traffic across multiple servers or service instances, preventing any single instance from becoming overloaded or becoming a single point of failure.
Contribution to HA:
- Fault Tolerance: If one server fails, the load balancer automatically redirects traffic to healthy servers — zero downtime for the user.
- Scalability: Instances can be added or removed based on traffic, keeping the system responsive under varying load.
- Traffic Distribution: Ensures no single component becomes a bottleneck, reducing the probability of failure from overloading.
Failover is the process by which a system automatically shifts to a backup system or component when the primary system fails. This ensures continuity of service during failures without requiring manual intervention.
How it helps:
- Minimizes Downtime: The system quickly switches to a secondary server, database, or resource without causing significant downtime.
- Improves Fault Tolerance: Ensures systems continue running even when components become unavailable.
- Example: In a web application, if the primary server goes down, a failover system immediately shifts traffic to a secondary server, allowing the service to continue without interruption.
Health Monitoring continuously checks the status of system components (servers, databases, services) to ensure they are functioning correctly. If a failure is detected, the system triggers recovery actions or alerts.
Self-Healing Systems automatically detect issues and take corrective actions without human intervention — restarting failed services, replacing unhealthy nodes, or scaling resources dynamically.
How they help:
- Proactive Fault Detection: Detect and address issues before they lead to significant failures.
- Automated Recovery: Reduces downtime by eliminating manual intervention lag.
- Example: AWS uses health checks to ensure EC2 instances are operational. If an instance fails, the system automatically replaces it with a new one, maintaining service availability.
System Reliability Questions
System reliability is the ability of a system to consistently perform its intended function without failure over a specified period. In system design, reliability ensures:
- Minimal downtime
- Consistent user experience
- Protection of data and transactions
- Trust in the system's behavior under stress or failure
High reliability is especially critical in distributed systems, mission-critical applications (banking, healthcare), and cloud-native environments where failure can cascade across services.
Availability means the system is accessible and operational when needed. Example: a website being online 24/7 with minimal downtime.
Durability refers to the ability to retain and preserve data without loss. Example: data written to AWS S3 is not lost even if a node fails.
Analogy: Think of availability as whether the ATM is working, and durability as whether your money is still in your account. You can have one without the other — an in-memory cache is fast and available but not durable; a cold backup tape is durable but not available during an outage.
MTBF (Mean Time Between Failures): Average time between two consecutive failures. Measures how reliable a system is — higher is better.
MTTR (Mean Time To Recovery): Average time to restore service after a failure. Measures resilience and repair efficiency — lower is better.
Relationship: Availability ≈ MTBF / (MTBF + MTTR). To maximize availability, you need failures to be rare (high MTBF) and recovery to be fast (low MTTR). Most teams underinvest in MTTR — cutting recovery time from 30 minutes to 2 minutes often improves availability more than preventing additional failures.
SLAs (Service Level Agreements) are formal contracts between service providers and customers that define:
- Expected uptime/availability (e.g., 99.9% uptime)
- MTTR and support response times
- Penalties for failure to meet guarantees (typically service credits)
SLAs create clear reliability goals, help prioritize engineering efforts, and hold teams accountable for reliability metrics. They also drive the SLO/SLI layering — the internal SLO target is always stricter than the SLA so that internal alarms fire before a customer contract is breached.
To achieve 99.99% ("four nines" — ~52 minutes of downtime per year):
- Redundancy: Multiple instances across availability zones and regions — no single node is indispensable.
- Load balancing and failover: Health checks pull failed instances from rotation automatically.
- Graceful degradation: Non-critical features shed load before core functionality degrades.
- Auto-healing: Auto-scaling groups restart or replace failed instances without manual intervention.
- Distributed storage: Multi-AZ databases (e.g., RDS Multi-AZ) with automated failover.
- Chaos engineering: Continuously test failure scenarios so you discover gaps before they become incidents.
Example: Multi-AZ setup in AWS with auto-scaling groups, health checks, circuit breakers, and retry-with-backoff at the application layer.
Systematic approach:
- Step 1: Check monitoring and alert logs for patterns — memory leaks, CPU spikes, OOM kills, GC pauses.
- Step 2: Analyze failure rate vs. expected MTBF to understand if failures are increasing in frequency.
- Step 3: Review recent code deployments and infrastructure changes — correlate with failure timestamps.
- Step 4: Add circuit breakers, retries with backoff, or bulkheads to improve resilience against downstream dependencies.
- Step 5: Add observability (distributed tracing, structured logging) to understand root cause at the request level.
Use SRE practices like blameless postmortems to identify systemic issues and prevent recurrence.
- Scale horizontally using stateless services so any instance can serve any request.
- Partition/shard databases to distribute data volume and query load.
- Use message queues (e.g., Kafka) to decouple services and absorb traffic spikes without cascading failures.
- Implement caching (e.g., Redis) to reduce repeated load on databases.
- Apply rate limiting and throttling to protect downstream services from being overwhelmed.
- Optimize MTTR with autoscaling groups and self-healing infrastructure.
Example stack: Kafka for event ingestion, Redis for caching, PostgreSQL read replicas for query load distribution.
- Deploy services in multiple availability zones or regions to tolerate data center failures.
- Use health checks with load balancers to reroute traffic away from unhealthy instances automatically.
- Leverage managed services (e.g., RDS Multi-AZ, DynamoDB global tables) that handle replication and failover internally.
- Automate failover using DNS-based routing tools like AWS Route 53 (geo DNS) or Azure Traffic Manager.
- Maintain replication and backup recovery plans tested regularly via DR drills.
Redundancy prevents single points of failure; failover ensures continuity when failures inevitably occur.
Sample answer: "On a real-time analytics dashboard, we initially fetched live data directly from the primary database to ensure up-to-the-second accuracy. However, this increased query load significantly and degraded reliability during traffic spikes. We chose to introduce a 15-second cache layer, accepting a slight staleness in the dashboard to greatly improve system stability and reduce MTTR during peak hours."
Framework for answering: Identify the trade-off explicitly (latency vs. consistency, throughput vs. durability), describe how you quantified the cost of each option, state what drove the decision (user impact, SLO budget, business priority), and explain how you monitored the outcome.
- Start with SLAs to define clear, bounded reliability goals — don't design for five nines when three nines meets the requirement.
- Apply the Pareto Principle: find the 20% of failure modes causing 80% of downtime and fix those first.
- Incremental improvement: monitor → analyze → fix one thing at a time rather than adding speculative infrastructure.
- Avoid unnecessary complexity: each extra layer (service mesh, sidecar, distributed cache) is a new failure mode — add only when the reliability benefit is measurable.
- Prefer managed cloud services where possible — let AWS/GCP handle the operational complexity of replication, backups, and failover.
Focus on simplicity, observability, and resilience over theoretical perfection.
Disaster Recovery Questions
Backup is the process of copying and storing data to protect against loss or corruption. Backups let you restore data to a previous state but do not ensure service availability during an outage — restoring from backup takes time.
Failover is automatic switching to a redundant or standby system when the primary fails. It keeps services running with minimal downtime but does not help if data has been corrupted or deleted.
Key distinction: Backups are for data recovery; failover is for service continuity. Best practice: use both — backups to restore lost or corrupted data, failover to keep systems live during infrastructure failures.
Step-by-step:
- Start with business requirements: Define RTO (Recovery Time Objective) and RPO (Recovery Point Objective) — these constrain every downstream decision.
- Multi-region deployment: Deploy application servers and databases in active-active or active-passive mode across regions. Replicate data asynchronously or synchronously based on RPO requirements.
- Implement failover: DNS-based failover (e.g., AWS Route 53, GCP Global LB) with health checks to detect failure and trigger routing changes automatically.
- Add data backups: Frequent snapshots, object versioning, and offsite storage to recover from data corruption or accidental deletion.
- Automate recovery: Runbooks and infrastructure-as-code so recovery is repeatable and fast.
- Perform DR drills: Regularly test the full recovery plan — an untested DR plan is not a DR plan.
RTO (Recovery Time Objective): The maximum acceptable time to restore service after a failure. If your RTO is 1 hour, the system must be serving traffic again within 60 minutes of an outage.
RPO (Recovery Point Objective): The maximum acceptable amount of data loss measured in time. If your RPO is 5 minutes, you cannot lose more than 5 minutes of writes.
How to optimize:
- Lower RTO: Automated failover, container orchestration (Kubernetes can reschedule pods in seconds), hot standby environments that are always ready to serve.
- Lower RPO: Real-time or near-real-time data replication — log shipping, change data capture (CDC), synchronous multi-region writes.
- Cost trade-off: Shorter RTO/RPO means higher infrastructure and operational cost. The goal is to meet business requirements, not to minimize both to zero.
- Data consistency: Keeping data in sync across regions is difficult, especially under network partition. Replication lag means the standby region may be behind the primary.
- Latency: Cross-region replication introduces delays — synchronous replication across continents adds significant write latency.
- Split-brain scenarios: Without proper coordination, different regions may each believe they are the primary, causing conflicting writes and data divergence.
- Data locality & compliance: Regulations like GDPR may restrict storing user data in certain regions, constraining where failover targets can be located.
- Failover orchestration complexity: Switching traffic, promoting standby databases, syncing in-flight transactions, and updating DNS all must happen in the right order during high-stress incidents.
Quorum-based design ensures that decisions — leader election, write confirmation, configuration changes — require agreement from a majority of nodes, preventing inconsistency or split-brain during failover.
Example: In a 5-node cluster, at least 3 nodes must agree (quorum = ⌊N/2⌋ + 1) before committing a write or electing a new leader. If 2 nodes are partitioned off, the remaining 3 still form a quorum and can proceed safely.
Used in:
- Consensus protocols: Paxos, Raft
- Systems: etcd, ZooKeeper, CockroachDB
Quorum ensures safe recovery and continued availability even during regional failures, at the cost of requiring a majority of nodes to be reachable for writes to proceed.
Further Reading
Interactive: Consistency Model Visualiser
Write a value to the primary, then immediately read from a replica. See how Strong, Eventual, and Read-Your-Writes consistency models behave differently under replication lag.