At a Glance
- Load balancers distribute traffic across servers to prevent overload.
- L4 (transport layer) routes by IP+port — fast, no content inspection.
- L7 (application layer) routes by URL/headers — smart, enables A/B testing and canary releases.
- Round robin is simplest; least-connections is smarter for heterogeneous request lengths.
- Active-active doubles capacity AND resilience; active-passive wastes standby capacity.
- Reverse proxies add SSL termination, compression, caching, and origin anonymity.
What a Load Balancer Does
A load balancer (LB) sits in front of your server fleet and spreads incoming requests across them. It's the traffic cop of horizontal scaling — and it does four jobs at once:
- Distribute requests so no single server drowns.
- Detect unhealthy servers and stop sending them traffic.
- Decouple clients from servers — you can add, remove, or replace servers invisibly.
- Defend — hide internal IPs, terminate TLS, absorb some abuse.
One front door, many workers: the LB fans requests out across the fleet.
Layer 4 vs Layer 7
The "layer" refers to the OSI network stack — and it determines what information the LB can see when routing.
Layer 4 (transport)
Sees only IPs and ports. Forwards TCP/UDP segments without reading the payload — basically high-speed network plumbing (often via NAT). Blazing fast, tiny CPU cost, but it can't tell /images from /checkout. Use for raw TCP/UDP services, databases, game servers, or as the outer tier at huge scale.
Layer 7 (application)
Terminates the connection and reads the actual request: URL, headers, cookies, hostname. Can route /api to one pool and /video to another, do TLS termination, compression, and per-user stickiness. Slightly more latency and CPU — almost always worth it for web traffic.
| Layer 4 | Layer 7 | |
|---|---|---|
| Routes on | IP + port | URL, headers, cookies, method |
| Speed | Fastest | Fast (parses requests) |
| Content-based routing | No | Yes |
| TLS termination | Pass-through | Yes |
| Examples | AWS NLB, LVS, HAProxy (TCP mode) | AWS ALB, Nginx, HAProxy (HTTP mode), Envoy |
Balancing Algorithms
Server 1, 2, 3, 1, 2, 3… Dead simple, great when servers and requests are uniform.
Bigger servers get proportionally more turns — useful in mixed-hardware fleets or for canary deploys (send 5% to the new version).
Send the next request to whichever server is handling the fewest right now. Shines when request durations vary wildly (some take 10 ms, some take 10 s).
Hash the client IP to pick a server — the same client consistently lands on the same box. A poor man's sticky session; struggles when many users share one NAT IP.
Pick two servers at random, send to the less-loaded one. Sounds silly, is mathematically brilliant — avoids both the herd behavior of "least connections" (everyone dog-piles the same idle server) and the blindness of pure random. Used by Envoy and Nginx.
Keeping the LB Itself Alive
If one load balancer fronts your fleet, congratulations: you've moved your single point of failure, not removed it. Two standard cures:
Active-passive LB pair
Two LBs share a floating IP (via VRRP/keepalived). The active one holds the IP and serves; the passive one watches heartbeats. If the active dies, the passive claims the IP in seconds. Drawback: half your LB capacity sits idle.
Active-active LB pair
Both LBs serve simultaneously — DNS round-robin or an upstream router (ECMP/anycast) spreads clients across them. Full hardware utilization, but each LB must be sized to carry 100% of traffic alone if its twin fails.
Health checks
The LB probes each backend every few seconds — a TCP connect (L4) or an HTTP GET /healthz expecting a 200 (L7). Fail N consecutive probes → ejected from rotation; pass M probes → welcomed back. Good health endpoints check real dependencies (DB reachable?) but stay cheap.
Session affinity (via cookie or IP hash) routes a user to the same server every time — handy if servers keep session state in local memory. But it skews load, and when that server dies its users get logged out. The cleaner fix: make servers stateless and park session state in Redis, so any server can answer any user.
Reverse Proxies: The LB's Multitalented Sibling
A reverse proxy is a server that accepts client requests on behalf of one or more backend servers — the mirror image of a forward proxy (which acts on behalf of clients). Even with a single backend, a reverse proxy earns its keep:
- TLS termination — handle certificates and crypto once, talk plain HTTP internally.
- Caching & compression — serve repeated responses and gzip/brotli everything.
- Security & anonymity — backends are never directly exposed; one place for WAF rules and rate limits.
- Static file serving — Nginx serves assets while proxying API calls to the app.
LB vs reverse proxy — what's actually the difference?
Mostly intent. A load balancer's purpose is distributing across many servers; a reverse proxy's purpose is fronting servers with extra capabilities, useful even with one backend. In practice the same software (Nginx, HAProxy, Envoy, Traefik) plays both roles, often in the same config file. AWS splits the roles into products: NLB (pure L4 balancing) vs ALB (L7 with routing rules).
Cloudflare is secretly "a reverse proxy for the whole internet" — CDN, WAF, TLS and DDoS protection are all reverse-proxy features at planetary scale. When an interviewer asks "where does TLS terminate?", the answer is almost always "at the L7 LB / reverse proxy."
Load Balancing Algorithms — A Visual Comparison
The steps above describe each algorithm; here's how the same stream of requests actually lands on a fleet of four servers under each policy — and where each one shines or stumbles.
1. Round Robin
1→2→3→4→1→2→3→4. Fair but dumb — it ignores that server 3 is busy grinding through a 10-second request.
2. Weighted Round Robin
Weights [3,2,1,1] produce the dispatch pattern 1,1,1,2,2,3,4. Good for heterogeneous hardware — the beefy box earns proportionally more turns.
3. Least Connections
Route to the server with the fewest open connections. Best for long-lived requests — WebSockets, streaming, slow report generation.
4. IP Hash
hash(client_IP) % N = shard. The same client always lands on the same server — required for sticky sessions, but the mapping breaks if a server is removed (N changes).
5. Random with Two Choices (R2C)
Pick 2 random servers, route to the one with fewer connections. Almost as good as least-connections at 10× less coordination overhead. Used by Nginx and HAProxy.
Global Server Load Balancing (GSLB)
A single-region LB handles failover within one datacenter. GSLB routes traffic across regions — US-East, EU-West, AP-Southeast — so users hit the closest healthy region and a regional outage doesn't take you offline.
- Methods: DNS-based (Route 53 latency routing), Anycast, or dedicated GSLB hardware (F5, Citrix).
- Health-aware: if US-East is down, the GSLB stops returning its IP — traffic shifts to EU-West within one DNS TTL.
- GeoDNS: EU users get EU IPs — reduces latency and satisfies GDPR data residency rules.
GSLB stops advertising the failed region — within one TTL, US-East traffic drains to EU-West.
Netflix's GSLB (Zuul + Route 53) combined with their Open Connect CDN means most users worldwide get sub-100ms time-to-first-byte. Latency-based DNS picks the region; Zuul routes within it.
Connection Draining and Graceful Shutdown
The problem: remove a server from rotation and you kill its active connections mid-request — users see errors for no reason. Connection draining fixes this: the LB stops sending new requests to the server but waits for in-flight requests to complete before removing it.
The LB flags the server: no new requests, existing connections stay alive.
The app stops accepting new connections and finishes what it has in hand.
AWS calls this "deregistration delay" — default 300s. Tune it to match your p99 request duration.
Once in-flight work is done (or the window expires), the process is killed and the server leaves the pool.
Rolling updates drain one instance while the others keep serving. No drain = every deploy drops a slice of live requests on the floor.
Health Checks — Knowing Who's Alive
Passive health checks
The LB simply observes real traffic. If a server returns 5xx three times in a row → mark unhealthy. Zero extra load, but detection is only as fast as your traffic.
Active health checks
The LB probes a dedicated endpoint (e.g., GET /health every 10s). Faster, deterministic detection — even for servers receiving little traffic.
Shallow vs deep checks
- Shallow check: just an HTTP 200 from the app. Fast and cheap, but doesn't catch "app up but DB down."
- Deep check: the app pings its DB, cache, and upstream services. Accurate — but dangerous: if the DB is merely slow, every app server's deep check fails at once → the LB removes all app servers → a partial slowdown becomes a system-wide outage.
- What to verify: can the app accept connections? Can it reach the DB? Can it reach its dependencies?
Use a shallow check for the LB (keep servers in rotation as long as they can possibly serve) and deep monitoring for alerting (wake a human when the DB path degrades). The LB's job is routing, not diagnosis.
Layer 4 vs Layer 7 — When to Use Each
The full decision matrix, feature by feature:
| Feature | L4 | L7 |
|---|---|---|
| Operates at | Transport (TCP/UDP/IP+port) | Application (HTTP/HTTPS/WebSocket) |
| Content inspection | None | Full headers, URL, body |
| Performance | Faster (less processing) | Slightly slower (parses requests) |
| SSL termination | Can passthrough or terminate | Terminates SSL (common choice) |
| Routing logic | IP:Port rules only | URL paths, hostnames, headers, cookies |
| Use case | Raw TCP apps, databases, SMTP | Web apps, APIs, microservices |
| Examples | AWS NLB, HAProxy TCP mode | AWS ALB, Nginx, Traefik, Envoy |
Real-World LB Stacks
Netflix (Zuul)
Custom L7 LB built on top of Nginx. Handles 100k+ RPS per instance. Routes by user ID to enable per-user A/B testing, and strips internal headers before forwarding to origin.
Cloudflare
Anycast-based global LB: every PoP accepts traffic on the same IPs, and BGP routes each client to the nearest one. Layer 7 with a WAF built in.
GitHub
HAProxy for L4 → Nginx for L7. Total throughput: tens of millions of git operations per day flowing through that two-tier stack.
Discord
Nginx → in-house Elixir services. WebSocket connections are routed by server (guild) ID at L7, so a guild's traffic lands on the node holding its state.
Reverse Proxy Deep Dive
A reverse proxy sits in front of origin servers — clients talk to the proxy, never to the origin directly. Beyond plain load balancing, it earns its keep six ways:
Decrypt HTTPS at the proxy, forward as HTTP internally. Origin CPU is freed from crypto work.
gzip/brotli response bodies at the proxy. Cuts bandwidth 60–80% for text content.
Cache static assets, full pages, or API responses at the proxy layer — requests never reach the app.
The proxy buffers slow client uploads before forwarding to the fast backend. App servers are protected from slow clients tying up worker threads.
Origin IPs are hidden. The proxy can block bad IPs, rate limit, and enforce auth before traffic touches your app.
Route a percentage of traffic to different backends based on headers or cookies — canary releases without app changes.
Nginx vs HAProxy vs Envoy
Nginx
Web server + reverse proxy + LB in one. Great for HTTP, static files, and SSL termination. The Swiss Army knife default.
HAProxy
Pure LB/proxy. Handles TCP and HTTP, powerful ACLs, and better raw L4 performance than Nginx.
Envoy
Modern proxy built for microservices and service mesh (Istio). First-class gRPC, HTTP/2, and distributed tracing support.
Interview Q&A
Three load-balancer questions interviewers actually ask — click to reveal strong answers.
This is the LB single point of failure problem. Solutions: (1) Active-passive LB pair with a floating VIP (virtual IP) — if the primary fails, the secondary takes over the VIP within seconds. (2) Anycast: multiple LB servers advertise the same IP, and BGP withdraws routes for failed ones automatically. (3) DNS round-robin to multiple LBs — cheap, but DNS caching means slow failover. Cloud-managed LBs (AWS ALB, GCP LB) have built-in redundancy — the provider handles it for you.
WebSockets are stateful, persistent connections, so the LB must: (1) Not break the TCP connection during the WebSocket upgrade. (2) Route the same connection to the same backend — sticky by connection, not just by cookie. (3) Handle very long-lived connections: default LB idle timeouts (60s, 120s) will silently kill WebSockets. Set the timeout to match the connection duration — hours or days for chat apps. Discord runs Nginx with upstream keepalive and proxy_read_timeout 86400 (24 hours).
(1) L4 for raw throughput — skip HTTP parsing entirely. (2) Kernel bypass networking: DPDK or eBPF to skip the OS network stack. (3) Consistent hashing for stateful routing. (4) Multiple LB instances behind Anycast or DNS round-robin. (5) Health checks via a separate control plane. Reference point: Cloudflare serves 46+ million requests/sec across their network using Anycast plus eBPF-based load balancing (their Unimog project).
Anti-Patterns to Avoid
Round robin for heterogeneous requests
"Add friend" takes 5ms; "generate report" takes 10 seconds. Round robin will pile reports onto random servers while others sit idle. Use least-connections instead.
Health checks that are too aggressive
Probe every 1s and remove on the first failure → constant flapping. Set a failure threshold (3 consecutive failures) and a recovery threshold (2 consecutive successes) to keep the pool stable.
Forgetting the LB itself is a SPOF
Teams add redundant app servers but front them with a single load balancer. Always make the LB redundant — it's the most critical component in the path.
Load Balancers & Proxies Interview Questions
All key questions and answers from this topic that appear in system design interviews.
Load Balancing Questions
Load balancing solves the problem of handling increasing traffic in a scalable and reliable manner. As the number of users grows, a single server can become overwhelmed by incoming requests. A load balancer distributes traffic across multiple servers, ensuring no single server carries all the workload. This improves system performance, enables scalability, and helps maintain availability even when individual servers experience issues.
Every server has finite resources: CPU, memory, storage, and network bandwidth. As user traffic increases, these resources become increasingly utilized until the server reaches its capacity limits. Once this happens, response times increase, requests begin to queue, and users experience timeouts or failures. Regardless of how powerful a server is, there is always a maximum capacity it can handle.
Vertical Scaling — Increasing the resources of an existing server: more CPU cores, memory, or storage. Simpler because the application remains on a single machine, but has physical and financial limits.
Horizontal Scaling — Adding more servers and distributing the workload across them. Requires additional infrastructure and traffic distribution mechanisms (load balancers) but provides significantly greater scalability and fault tolerance — the preferred approach for large-scale systems.
Horizontal scaling provides a more sustainable path for long-term growth. While vertical scaling can temporarily increase capacity, there is always a limit to how much a single server can be upgraded, and larger servers become increasingly expensive. By adding more servers, organizations can continue increasing capacity as demand grows. Horizontal scaling also improves fault tolerance because the system can continue operating even if one server fails.
Once multiple application servers are introduced, incoming requests must be distributed among them efficiently. Users should not need to know which server to connect to, and traffic should be directed in a way that makes effective use of available resources. Without a mechanism to manage this distribution, some servers may become overloaded while others remain underutilized. This challenge creates the need for a load balancer.
A load balancer acts as a single entry point for incoming requests. Instead of clients connecting directly to individual application servers, they send requests to the load balancer, which then forwards those requests to one of the available backend servers. This abstraction simplifies client interactions, enables horizontal scaling, and allows the system to distribute traffic efficiently. It also provides a foundation for improving availability and reliability within distributed architectures.
Load balancing improves availability by reducing dependency on any single server. When multiple servers are available to handle requests, the system can continue operating even if one server becomes unavailable. This redundancy minimizes downtime and ensures users can continue accessing the application despite infrastructure failures, maintenance activities, or unexpected outages.
In a load-balanced architecture, a failed server can be removed from active traffic routing, allowing requests to be directed to the remaining healthy servers. The application continues serving users with minimal disruption. A single server failure no longer causes the entire application to become unavailable — the impact of failures is isolated, making the system more resilient.
Load balancing enables many characteristics expected from modern applications: scalability, high availability, reliability, and fault tolerance. As systems grow beyond a single server, a mechanism is needed to distribute traffic and manage multiple instances. Load balancing provides this capability and serves as a critical building block for distributed systems, cloud-native architectures, and large-scale applications.
A load balancer should be introduced when a single server can no longer meet performance, scalability, or availability requirements. This typically occurs when traffic increases significantly, when multiple application servers are required, or when the business demands higher uptime and reliability. Introducing a load balancer becomes especially valuable when adopting horizontal scaling strategies.
Most applications begin with a single server — simple and cost-effective. As traffic grows, that server reaches its capacity limits, leading to performance issues and reduced reliability.
The first response is often vertical scaling by upgrading the server's resources. However, this approach eventually reaches practical limits.
To continue growing, additional application servers are introduced through horizontal scaling. Once multiple servers exist, traffic must be distributed among them efficiently. A load balancer is added in front of the servers to act as a single entry point and route requests appropriately. This evolution enables the system to handle larger workloads, improve availability, and become more resilient to failures.
Load balancing improves scalability by allowing traffic to be distributed across multiple servers rather than relying on a single machine. As traffic increases, additional servers can be added to the system, and the load balancer can begin routing requests to them. This approach enables capacity to grow incrementally and helps organizations handle increasing workloads without requiring major architectural changes or constant hardware upgrades.
Forward Proxy vs. Reverse Proxy Questions
A proxy server is an intermediary system that sits between a client and a destination server. When a client makes a request, the proxy server forwards it to the destination, receives the response, and relays it back to the client.
Security & Privacy — Hides the client's or server's identity by masking IP addresses.
Caching & Performance — Frequently requested content is stored and served faster.
Traffic Control & Load Balancing — Helps distribute traffic evenly across multiple servers.
Content Filtering — Blocks access to restricted or harmful content.
Compression & Optimization — Reduces bandwidth consumption.
Forward Proxy — Sits between clients and the internet. The client connects to the forward proxy, which forwards the request to the target server. Used by clients to access external resources securely or anonymously. Common uses: hiding user identity, bypassing geo-restrictions, caching content. Examples: Squid Proxy, VPNs, Tor.
Reverse Proxy — Sits between users and backend servers. The user connects to the reverse proxy, which forwards the request to an appropriate backend server. Used by servers to manage, secure, and optimize incoming traffic. Common uses: load balancing, caching, SSL termination, DDoS protection. Examples: Nginx, HAProxy, Cloudflare, AWS Elastic Load Balancer.
Key distinction: Forward proxy protects clients; reverse proxy protects servers.
Hiding Client Identity — A forward proxy masks the client's IP address so the target website only sees the proxy's IP. This enables anonymous browsing and bypassing geo-blocking restrictions.
Encryption of Traffic — VPN-based forward proxies encrypt internet traffic, preventing ISPs and hackers from spying on user activities.
Content Filtering & Malware Protection — Organizations use forward proxies to block malicious websites and prevent phishing attacks.
Access Control — Companies restrict employee access to specific websites using forward proxies.
Load Balancing — A reverse proxy distributes incoming requests across multiple backend servers using algorithms such as Round Robin, Least Connections, and IP Hashing.
Caching — Frequently requested content (HTML pages, images, videos) is stored in the reverse proxy's cache. Subsequent requests are served from cache rather than the backend, reducing response time and server load.
Security & DDoS Protection — Reverse proxies block malicious traffic before it reaches backend servers. Example: Cloudflare protects websites from DDoS attacks by filtering traffic at the proxy level.
Use a forward proxy when: clients need anonymous or secure access to external resources (e.g., VPNs, corporate web filtering, bypassing geo-restrictions, blocking access to certain websites).
Use a reverse proxy when: backend servers need protection, load balancing, caching, or SSL termination (e.g., CDN services, API gateways, high-traffic web applications).
Both types can be used for caching content to improve performance.
Traffic Filtering — Analyzes requests and blocks malicious IPs and bots before they reach backend servers.
Rate Limiting — Limits the number of requests from a single IP to prevent excessive traffic.
Anomaly Detection — Uses AI-based traffic analysis to detect attack patterns.
Load Distribution — Distributes traffic across multiple servers to prevent overload during high-traffic attacks.
SSL termination is the process where the reverse proxy decrypts HTTPS traffic before forwarding it to backend servers (which then communicate over plain HTTP internally).
Benefits: Reduces backend server CPU load (no SSL decryption needed), improves performance with faster response times, and provides centralized SSL certificate management at the proxy level. Example: Cloudflare handles SSL termination for websites, reducing the burden on origin servers.
Forward Proxy Examples: VPN Services (NordVPN, ExpressVPN) mask user identity and allow access to blocked websites; Corporate Web Filters restrict access to non-work-related websites; Tor Network provides anonymous browsing.
Reverse Proxy Examples: Cloudflare CDN acts as a reverse proxy to optimize website performance and security; Nginx handles high web traffic loads efficiently; AWS Elastic Load Balancer distributes incoming requests across multiple cloud servers.
Cloudflare — DDoS protection, CDN caching, global load balancing, and managed SSL. Best for internet-facing services needing global edge presence.
Nginx — High-performance web server and reverse proxy, easy to configure, great for static content caching and microservices. Best for self-managed infrastructure.
HAProxy — Enterprise-grade load balancing with advanced health checks, session persistence, and high availability. Best for high-throughput, low-latency L4/L7 balancing.
Further Reading
Quick Check
You need to route /api/* to your API fleet and /static/* to a file server pool. What's the minimum you need?
Interactive: Load Balancing Algorithms
Switch between Round Robin, Least Connections, Random, and Weighted algorithms. Click "Send Request" to route traffic manually, or use "Send 10" to see distribution patterns emerge. Notice how Least Connections adapts to uneven server load.