Computer Networks · The Transport Layer

TCP in the Real World

JrCodex9 min readChapter 19 of 36

Jr Codex Computer Networks Notes

Level: Advanced Prerequisites: Chapter 4: Flow and Congestion Control Time to complete: ~20 minutes

Table of Contents

  1. Nagle and Delayed ACK
  2. Keepalives
  3. Head-of-Line Blocking
  4. TCP on Wireless
  5. QUIC
  6. Tuning Checklist
  7. Summary & Next Steps

1. Nagle and Delayed ACK

NAGLE'S ALGORITHM
─────────────────────────────────────────
  PROBLEM: a telnet session sends one byte per
  keystroke. With 40 bytes of headers (Module 1,
  Chapter 3), efficiency is 2.4%.

  RULE: if there is unacknowledged data outstanding,
  BUFFER small writes until either an ACK arrives
  or a full segment's worth accumulates.

  Result: many small writes are coalesced into
  fewer, larger segments.

DELAYED ACK
─────────────────────────────────────────
  PROBLEM: sending an ACK for every segment doubles
  the packet count.

  RULE: wait up to 200ms before acknowledging, in
  the hope of either having data to send back
  (piggybacking the ACK) or of a second segment
  arriving to acknowledge both at once.

THE INTERACTION — a 200ms stall
─────────────────────────────────────────
  Individually sensible. Together, pathological.

    Application does a small write, then another.
    Nagle holds the second write, waiting for an
    ACK of the first.
    The receiver holds the ACK, waiting for data to
    piggyback on or a second segment.

    Deadlock, broken only by the 200ms delayed-ACK
    timer.

  SYMPTOM: a request-response protocol that is
  consistently ~200ms slower than the network
  justifies, with no packet loss and low latency.

  The classic case is a write of the header
  followed by a write of the body.
─────────────────────────────────────────

import socket
 
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)   # DISABLE Nagle
 
# Better still: avoid the problem rather than the algorithm.
# ✗ two writes ──► Nagle may hold the second
s.sendall(header); s.sendall(body)
 
# ✓ ONE write ──► nothing to coalesce, no interaction possible
s.sendall(header + body)
The Guidance
─────────────────────────────────────────
  Set TCP_NODELAY for interactive and
  request-response protocols — most RPC libraries
  and HTTP servers already do.

  But writing complete messages in ONE call is the
  better fix: it avoids the interaction entirely
  AND reduces syscalls.
─────────────────────────────────────────

2. Keepalives

The Problem
─────────────────────────────────────────
  An idle TCP connection sends nothing. If the peer
  crashes, or a cable is unplugged, NOTHING NOTIFIES
  YOU.

  The connection sits in ESTABLISHED on your side
  forever — a HALF-OPEN connection. You discover it
  only when you try to send.

  Worse, NAT and firewall state (Module 3,
  Chapter 5) expires after a few minutes of
  silence. The mapping is discarded, and your next
  packet is dropped or reset.
─────────────────────────────────────────

import socket
 
def enable_keepalive(sock, idle=60, interval=10, count=3):
    """Detect dead peers, and hold NAT mappings open."""
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
    sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, idle)      # idle before probe
    sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPINTVL, interval) # between probes
    sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPCNT, count)      # probes before
                                                                        # declaring dead
The Default Is Useless
─────────────────────────────────────────
  Linux defaults to TCP_KEEPIDLE of 7200 seconds —
  TWO HOURS before the first probe.

  Every NAT device in the path will have discarded
  the mapping long before that.

  So SO_KEEPALIVE alone does nothing useful. You
  must set the intervals, and 30-60 seconds is the
  usual choice.

  APPLICATION-LEVEL heartbeats are often better
  still: they verify the application is alive, not
  merely the kernel, and they work through
  middleboxes that filter TCP keepalives.
─────────────────────────────────────────

3. Head-of-Line Blocking

The Problem
─────────────────────────────────────────
  TCP guarantees ORDERED delivery of the whole byte
  stream.

  So if segment 5 of 100 is lost, segments 6-100
  may all have ARRIVED — and TCP will not deliver
  ANY of them to the application until 5 is
  retransmitted and received.

  Data is sitting in the receiver's buffer, correct
  and complete, and unavailable.
─────────────────────────────────────────

Why HTTP/2 Made It Worse
─────────────────────────────────────────
  HTTP/1.1 opened SIX TCP connections per origin.
  A loss on one blocked only that connection.

  HTTP/2 multiplexes ALL requests over ONE
  connection (Module 5, Chapter 3) — better for
  congestion control and connection overhead.

  But now ONE lost packet blocks EVERY concurrent
  request, because they share one ordered byte
  stream.

  HTTP/2 solved application-layer head-of-line
  blocking and inherited it at the transport layer.
  Solving that required replacing TCP — which is
  what QUIC did.
─────────────────────────────────────────

4. TCP on Wireless

The Broken Assumption
─────────────────────────────────────────
  Congestion control (Chapter 4) assumes LOSS MEANS
  CONGESTION. On wired networks that is almost
  always true.

  On wireless, loss is frequently CORRUPTION —
  interference, a wall, a moving device (Module 2,
  Chapter 1).

  TCP cannot tell them apart, so it halves its
  window for a problem that has nothing to do with
  congestion. The link is idle and TCP has
  throttled itself.

  This is the mechanism behind Module 1, Chapter
  5's finding that 1% loss can halve throughput.
─────────────────────────────────────────

The Mitigations
─────────────────────────────────────────
  LINK-LAYER RETRANSMISSION
    Wifi retries lost frames at layer 2 (Module 2,
    Chapter 3), so TCP never sees most corruption
    loss. This is the main reason TCP works on
    wifi at all.
    COST: retries take time, so RTT becomes highly
    variable — which is why the RTO variance term
    (Chapter 3) matters so much here.

  BBR (Chapter 4)
    Does not use loss as its signal, so random
    corruption does not trigger a reduction.

  FORWARD ERROR CORRECTION at the link layer
    (Module 2, Chapter 2)

  MOBILITY
    Changing cell or access point changes your IP,
    which KILLS every TCP connection — because a
    connection is identified by its 4-tuple
    (Chapter 1). QUIC fixes this.
─────────────────────────────────────────

5. QUIC

What It Is
─────────────────────────────────────────
  A transport protocol built on UDP, implemented in
  USERSPACE, standardised in 2021. It is the
  transport under HTTP/3.

  It reimplements TCP's guarantees and fixes four
  of TCP's structural problems — and it could only
  do so by not being TCP.
─────────────────────────────────────────

The Four Fixes
─────────────────────────────────────────
  1. NO HEAD-OF-LINE BLOCKING
     Multiple independent STREAMS within one
     connection. Loss on stream 3 blocks only
     stream 3; the others deliver normally.
     ── the direct fix for Section 3.

  2. FASTER HANDSHAKE
     TCP handshake + TLS handshake = 2-3 round
     trips (Chapter 3). QUIC combines transport and
     crypto setup into ONE round trip, and ZERO for
     a resumed connection.

  3. CONNECTION MIGRATION
     A connection is identified by a CONNECTION ID,
     not the 4-tuple. Change from wifi to mobile,
     your IP changes, and the connection SURVIVES.
     ── the direct fix for Section 4's mobility
        problem.

  4. DEPLOYABLE EVOLUTION
     Running in userspace over UDP means
     improvements ship with the application, not
     with an OS kernel update. TCP changes take a
     decade to deploy; QUIC changes take a release.
─────────────────────────────────────────

Why It Had to Be UDP
─────────────────────────────────────────
  Middleboxes — NATs, firewalls, "optimisers" —
  inspect and modify TCP. A genuinely new transport
  protocol number would be dropped by much of the
  internet (this is protocol OSSIFICATION).

  UDP passes everywhere. So QUIC hides inside UDP,
  and encrypts almost its entire header so
  middleboxes cannot inspect or interfere with it
  — deliberately preventing the ossification that
  trapped TCP.
─────────────────────────────────────────

The Costs
─────────────────────────────────────────
  - Higher CPU than TCP: no decades of kernel and
    NIC offload optimisation, though this is
    improving
  - Some networks block or throttle UDP
  - Harder to inspect operationally, by design
  - Encrypted headers mean traditional network
    troubleshooting tools see much less
─────────────────────────────────────────

6. Tuning Checklist

APPLICATION LEVEL — where most wins are
─────────────────────────────────────────
  □ REUSE CONNECTIONS. Keep-alive and connection
    pooling avoid the handshake (Chapter 3) and
    slow start (Chapter 4) on every request. This
    is usually the single biggest improvement.
  □ Write complete messages in ONE sendall
    (Section 1)
  □ Set TCP_NODELAY for request-response protocols
  □ Configure keepalive intervals; do not rely on
    defaults (Section 2)
  □ Set explicit timeouts on every socket

SYSTEM LEVEL
─────────────────────────────────────────
  □ tcp_window_scaling = 1     (Chapter 4)
  □ tcp_sack = 1               (Chapter 3)
  □ congestion control: cubic, or bbr for
    high-BDP or lossy paths
  □ increase socket buffer maxima for high-BDP
    paths
  □ raise the listen backlog on busy servers
    (Chapter 1)

NETWORK LEVEL
─────────────────────────────────────────
  □ Verify the path MTU; do not block ICMP
    (Module 3, Chapter 5)
  □ Enable fq_codel or CAKE to control bufferbloat
    (Chapter 4)
  □ Reduce DISTANCE — a CDN attacks the one
    component of latency nothing else can touch
    (Module 7, Chapter 2)

The Ordering Principle
─────────────────────────────────────────
  Fix the APPLICATION first. Connection reuse and
  fewer round trips beat every kernel parameter,
  usually by a wide margin.

  Kernel tuning matters at the extremes —
  high-BDP paths, very high connection rates. It is
  rarely where an ordinary slow application's
  problem lies.
─────────────────────────────────────────

7. Summary & Next Steps

Key Takeaways

  • Nagle and delayed ACK are individually sensible and interact to produce a 200ms stall in request-response protocols; writing complete messages in one call avoids it entirely.
  • TCP keepalive defaults to two hours, long after every NAT mapping has expired, so the intervals must be set explicitly or application heartbeats used.
  • Head-of-line blocking means one lost segment stalls all data behind it, and HTTP/2's single-connection multiplexing made this worse rather than better.
  • QUIC fixes head-of-line blocking, handshake latency, connection migration and protocol ossification — and had to hide inside UDP to be deployable at all.

Module 4 Complete — What's Next

You now have a reliable, congestion-controlled byte stream between two programs. Module 5 covers what those programs actually say to each other: DNS, HTTP, and the protocols that make up the web.

Concept Check

  1. Explain the Nagle and delayed-ACK interaction, and give two different fixes.
  2. Why did HTTP/2's single connection make transport-layer head-of-line blocking worse?
  3. Which two of QUIC's four improvements directly address problems this module identified in TCP, and what are they?

Next Module

→ Module 5: The Application Layer

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to Computer Networks Index