Modern And Distributed Networking
Load Balancing and Reverse Proxies
AVAILABILITY one machine is one failure domain
JrCodex·9 min read
Jr Codex Computer Networks Notes
Level: Advanced Prerequisites: Module 6, Chapter 4: Firewalls, VPNs and Defence in Depth Time to complete: ~20 minutes
Table of Contents
- Why Load Balancers Exist
- Layer 4 vs Layer 7
- Balancing Algorithms
- Health Checks
- Sticky Sessions
- The Load Balancer as a Single Point of Failure
- Summary & Next Steps
1. Why Load Balancers Exist
Two Problems, One Solution
─────────────────────────────────────────
CAPACITY one machine has a ceiling
AVAILABILITY one machine is one failure domain
A load balancer distributes requests across
several backends, and stops sending to any that
fail.
─────────────────────────────────────────
What Else It Provides
─────────────────────────────────────────
TLS TERMINATION decrypt once at the edge
(Module 6, Chapter 3) rather
than on every backend
A SINGLE ADDRESS backends may be added and
removed without any DNS change
ROUTING by path, header or host
(layer 7)
RATE LIMITING and DDoS absorption
(Module 6, Chapter 4)
ZERO-DOWNTIME drain a backend, deploy,
DEPLOYS return it
─────────────────────────────────────────
Reverse vs Forward Proxy
─────────────────────────────────────────
FORWARD PROXY sits in front of CLIENTS.
The client knows about it; the
server does not.
Corporate egress filtering,
caching proxies.
REVERSE PROXY sits in front of SERVERS.
The client thinks it IS the
server.
A load balancer is a reverse
proxy.
Same machinery, opposite direction, different
purpose.
─────────────────────────────────────────
2. Layer 4 vs Layer 7
LAYER 4 — transport
─────────────────────────────────────────
Balances on IP and PORT only. Forwards TCP
segments without inspecting the payload.
+ very fast; can operate at line rate
+ protocol-agnostic — works for anything over TCP
+ TLS passes through UNTOUCHED, so it never
decrypts
- cannot route by URL, header or cookie
- cannot retry a failed request — it does not
know what a request is
LAYER 7 — application
─────────────────────────────────────────
Terminates the connection, parses HTTP, and makes
a NEW connection to the backend.
+ route by path, host, header, cookie
+ RETRY a failed request on another backend
+ rewrite headers, compress, cache
+ one connection from the client can be
multiplexed to many backends
- slower; must parse
- must TERMINATE TLS to see anything
- it is now a participant, not a forwarder
─────────────────────────────────────────
DIRECT SERVER RETURN
─────────────────────────────────────────
A layer 4 trick: the load balancer forwards the
REQUEST, and the backend replies DIRECTLY to the
client, bypassing the balancer.
Why it helps: responses are typically far larger
than requests. Removing them from the balancer's
path removes most of its bandwidth load.
Why it is rare now: it requires layer 2 adjacency
and careful configuration, and it is incompatible
with TLS termination.
─────────────────────────────────────────
Choosing
─────────────────────────────────────────
Need to route by URL or header, or to retry
──► LAYER 7. Almost always the answer for
HTTP.
Extreme throughput, non-HTTP protocols, or TLS
must reach the backend untouched
──► LAYER 4.
Common architecture: layer 4 at the very edge for
volume and DDoS absorption, layer 7 behind it for
routing.
─────────────────────────────────────────
3. Balancing Algorithms
The Options
─────────────────────────────────────────
ROUND ROBIN
Each backend in turn. Simple.
Assumes all requests cost the same and all
backends are identical. Both are usually false.
WEIGHTED ROUND ROBIN
Proportional to capacity. Handles heterogeneous
hardware.
LEAST CONNECTIONS
Send to whichever backend has fewest active
connections.
── adapts automatically when requests vary in
duration, which they always do
LEAST RESPONSE TIME
Combines connection count with observed
latency. Reacts to a backend that is degraded
but not failed.
IP HASH
hash(client IP) picks the backend.
Deterministic, so the same client returns to
the same backend — poor man's stickiness
(Section 5).
CONSISTENT HASHING
As in DBMS Notes, Module 9, Chapter 3. Adding
or removing a backend remaps only ~1/N of
clients rather than nearly all of them.
── essential when the backend holds a CACHE
─────────────────────────────────────────
import bisect, hashlib
class ConsistentHashBalancer:
"""Adding a backend remaps ~1/N of keys, not all of them."""
def __init__(self, backends, vnodes=160):
self.ring, self.keys = {}, []
for b in backends:
self.add(b, vnodes)
def _hash(self, s):
return int(hashlib.md5(s.encode()).hexdigest(), 16)
def add(self, backend, vnodes=160):
for i in range(vnodes): # VIRTUAL NODES keep the load even
h = self._hash(f"{backend}:{i}")
self.ring[h] = backend
bisect.insort(self.keys, h)
def pick(self, key):
h = self._hash(key)
return self.ring[self.keys[bisect.bisect(self.keys, h) % len(self.keys)]]THE POWER OF TWO CHOICES
─────────────────────────────────────────
Pick TWO backends at random; send to whichever
has fewer connections.
This performs almost as well as checking ALL
backends, at a fraction of the coordination cost
— and dramatically better than picking one at
random.
It is the default in several modern proxies, and
it is a genuinely surprising result: two samples
buy nearly all the benefit of global knowledge.
─────────────────────────────────────────
4. Health Checks
Two Kinds
─────────────────────────────────────────
ACTIVE
The balancer periodically probes each backend.
Detects failure before a user does.
Costs a small constant load.
PASSIVE
Observe real traffic. Too many errors or
timeouts ──► mark unhealthy.
No extra load, and users experience the first
failures.
USE BOTH. Active catches a backend that is down;
passive catches one that is up and broken.
─────────────────────────────────────────
SHALLOW vs DEEP
─────────────────────────────────────────
SHALLOW GET /healthz ──► 200 if the process is
running.
Cheap. Says nothing about whether the
service WORKS.
DEEP Checks dependencies: can it reach the
database, the cache, the queue?
Meaningful, and DANGEROUS.
─────────────────────────────────────────
WHY DEEP CHECKS CAUSE OUTAGES
─────────────────────────────────────────
The database has a brief problem.
Every backend's deep health check fails
simultaneously. The load balancer removes ALL of
them. The service is now completely down —
including for requests that never touch the
database.
A degraded dependency became a total outage,
caused by the health check.
THE RIGHT DESIGN
LIVENESS is the process alive? Shallow. If
this fails, RESTART it.
READINESS can it serve traffic? Slightly
deeper. If this fails, stop sending
traffic — but do not restart.
And never let a shared dependency's failure
mark every instance unhealthy at once.
─────────────────────────────────────────
upstream backend {
least_conn;
server 10.0.1.10:8080 max_fails=3 fail_timeout=30s; # PASSIVE
server 10.0.1.11:8080 max_fails=3 fail_timeout=30s;
server 10.0.1.12:8080 backup; # used only if others fail
keepalive 32; # reuse backend connections
}
server {
listen 443 ssl http2;
location / {
proxy_pass http://backend;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_next_upstream error timeout http_502 http_503; # RETRY elsewhere
proxy_connect_timeout 2s;
proxy_read_timeout 30s;
}
}X-Forwarded-For, and Why It Matters
─────────────────────────────────────────
Once a layer 7 balancer terminates the
connection, the backend sees the BALANCER's IP,
not the client's.
X-Forwarded-For carries the original address —
needed for logging, rate limiting and geolocation.
SECURITY NOTE: a client can forge this header.
Trust it ONLY from your own proxies, and strip
or overwrite anything arriving from outside.
─────────────────────────────────────────
5. Sticky Sessions
The Problem
─────────────────────────────────────────
A backend holds per-user state in memory. If the
next request lands on a different backend, the
state is gone.
STICKINESS routes a user consistently to the same
backend — by cookie, or by IP hash.
─────────────────────────────────────────
Why It Is a Workaround
─────────────────────────────────────────
It reintroduces the problems statelessness
removed:
- UNEVEN LOAD: sticky users cluster
- LOST STATE on backend failure — the user is
logged out
- DEPLOYS disrupt every session on a restarted
instance
- SCALING OUT does not help existing sessions
THE PROPER FIX: make backends STATELESS. Put
session state in a shared store (Redis) or in a
signed token (Module 5, Chapter 2).
Then ANY backend serves ANY request, stickiness
is unnecessary, and failure or deployment of one
instance is invisible.
Use stickiness when you cannot change the
application — and treat it as debt.
─────────────────────────────────────────
6. The Load Balancer as a Single Point of Failure
The Irony
─────────────────────────────────────────
You added a load balancer for availability. Now
everything depends on it.
─────────────────────────────────────────
The Layers of Redundancy
─────────────────────────────────────────
DNS ROUND ROBIN
Several A records for one name. Clients pick
one, and retry another on failure.
Crude — DNS caching means slow failover — but
it removes the single address.
ANYCAST
ONE IP announced from many locations (Module 3,
Chapter 4). BGP routes each client to the
nearest, and a failed site simply withdraws its
announcement.
── how large providers do it
ACTIVE-PASSIVE PAIR
Two balancers sharing a virtual IP; the standby
takes over on failure.
MULTIPLE ACTIVE
Several balancers, all serving, behind anycast
or DNS.
─────────────────────────────────────────
The Principle
─────────────────────────────────────────
Every layer that improves availability becomes a
dependency that can fail.
The answer is never one perfect component. It is
redundancy at each layer, and failure modes that
DEGRADE rather than collapse.
Ask of any component you add: "what happens when
THIS fails?" — and make sure the answer is not
"everything stops".
─────────────────────────────────────────
7. Summary & Next Steps
Key Takeaways
- Layer 4 balancing forwards without inspecting and cannot retry; layer 7 terminates the connection, enabling routing by URL and retrying failed requests elsewhere.
- Least connections adapts to varying request durations automatically, and the power of two random choices achieves nearly the same result with far less coordination.
- Deep health checks turn a degraded shared dependency into a total outage by marking every instance unhealthy at once — separate liveness from readiness.
- Sticky sessions are a workaround that reintroduces the problems statelessness solved; putting session state in a shared store is the actual fix.
Concept Check
- Why can a layer 7 load balancer retry a failed request while a layer 4 one cannot?
- Explain how a deep health check can convert a slow database into a complete outage.
- Why is
X-Forwarded-Foruntrustworthy when it arrives from outside your infrastructure?
Next Chapter
→ Chapter 2: CDNs and Edge Caching
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to Computer Networks Index