Computer Networks

Practice And Capstone

Debugging Methodically

Changing three and finding it works tells you

JrCodex·9 min read

Jr Codex Computer Networks Notes

Level: Advanced Prerequisites: Chapter 1: The Diagnostic Toolkit Time to complete: ~20 minutes

Table of Contents

  1. The Method
  2. Narrowing Before Testing
  3. Six Worked Symptoms
  4. Intermittent Problems
  5. Common Wrong Turns
  6. A Checklist
  7. Summary & Next Steps

1. The Method

The Loop
─────────────────────────────────────────
  1. DESCRIBE the symptom precisely. Not "it is
     slow" — "requests to /api/orders take 8
     seconds, from every client, since 14:00
     yesterday."

  2. NARROW the scope (Section 2). Which clients,
     which destinations, which times, which
     protocols?

  3. TEST BOTTOM-UP. The first failing layer is
     where to look (Module 1, Chapter 2).

  4. FORM ONE HYPOTHESIS, and a test that would
     DISPROVE it.

  5. CHANGE ONE THING. Measure. Revert if it did
     not help.

  6. Repeat.
─────────────────────────────────────────

The Two Rules That Save the Most Time
─────────────────────────────────────────
  CHANGE ONE THING AT A TIME.
    Changing three and finding it works tells you
    nothing about which mattered — and leaves two
    unnecessary changes in production.

  WRITE DOWN WHAT YOU TESTED AND WHAT HAPPENED.
    In a long investigation you WILL forget whether
    you already ruled something out, and re-test
    it. Notes turn thrashing into progress.
─────────────────────────────────────────

2. Narrowing Before Testing

The Five Questions
─────────────────────────────────────────
  WHO?    all users, or some?
          one region, one office, one ISP?
          one client version?

  WHAT?   all traffic, or one service?
          all requests, or only large ones?
          (── large ones only points at MTU:
              Module 1, Chapter 3)

  WHERE?  from inside the network, or outside?
          from one datacentre, or all?

  WHEN?   always, or at peak?
          (── at peak means CONTENTION, not a
              broken path)
          since a specific change?

  HOW BAD? fails entirely, or is slow?
           what percentage?
─────────────────────────────────────────

Why Narrowing Beats Testing
─────────────────────────────────────────
  "Only users in one office, only since Tuesday,
   only for uploads over 1 MB"

  has already told you more than an hour of
  packet captures would. It points at that
  office's path, a change on Tuesday, and an MTU
  or timeout issue.

  Most of the diagnostic work is DESCRIBING the
  problem accurately. The tools then confirm a
  hypothesis rather than searching blindly.
─────────────────────────────────────────

3. Six Worked Symptoms

1. "THE SITE DOES NOT LOAD"
─────────────────────────────────────────
  ping 8.8.8.8       fails ──► no connectivity at
                     all. Check layer 1-2.
                     works ──► continue

  dig example.com    fails ──► DNS. Compare with
                     the authoritative server
                     (Chapter 1).

  nc -zv host 443    refused ──► nothing listening
                     timeout ──► firewall or
                     routing

  curl -v            TLS error ──► certificate,
                     protocol version, or clock
                     skew
                     HTTP error ──► the application
─────────────────────────────────────────

2. "IT WORKS FOR SMALL REQUESTS, HANGS ON LARGE"
─────────────────────────────────────────
  ──► PATH MTU DISCOVERY IS BROKEN.

  Almost certainly ICMP is blocked somewhere
  (Module 1, Chapter 3; Module 3, Chapter 5).

  CONFIRM:
    ping -M do -s 1472 host      # succeeds?
    ping -M do -s 1400 host      # succeeds where
                                 # 1472 failed?

  FIX: unblock ICMP type 3 code 4, or clamp the
  MSS at the tunnel or firewall.

  This symptom has ONE common cause. Recognising it
  saves hours.
─────────────────────────────────────────

3. "SLOW ONLY AT PEAK TIMES"
─────────────────────────────────────────
  ──► CONTENTION, not a broken path.

  CHECK:
    - queuing delay: does ping RTT rise at peak?
      (Module 1, Chapter 5)
    - is a large transfer filling a buffer?
      BUFFERBLOAT (Module 4, Chapter 4)
    - connection or thread pool exhaustion at the
      application
    - a shared upstream at its limit

  A path that is fine off-peak is not misrouted.
─────────────────────────────────────────

4. "INTERMITTENT CONNECTION RESETS"
─────────────────────────────────────────
  ──► something is actively closing connections.

  CANDIDATES:
    - a NAT or firewall IDLE TIMEOUT killing
      long-lived connections (Module 4, Chapter 5)
      ── fix with keepalives
    - a load balancer idle timeout shorter than
      the client's
    - an application crash or restart
    - a connection limit being hit

  CONFIRM with tcpdump: WHO sends the RST, and how
  long after the last data?
─────────────────────────────────────────

5. "FAST FROM THE OFFICE, SLOW FROM HOME"
─────────────────────────────────────────
  ──► a PATH difference, not a server problem.

    mtr from both, and compare
    ── where do the paths diverge, and where does
       latency or loss appear?

  Also consider: is the office hitting a cache or
  an internal route that home users do not?

  If ttfb is identical from both (Chapter 1), the
  SERVER is equally fast for both and the
  difference is entirely network.
─────────────────────────────────────────

6. "IT WORKS FROM MY MACHINE BUT NOT FROM THE
    CONTAINER"
─────────────────────────────────────────
  THE USUAL CAUSES, in order of frequency:
    - the service is bound to 127.0.0.1 rather than
      0.0.0.0 (Chapter 1)
    - the container's DNS resolver differs
    - a network namespace or bridge issue
    - a security group or firewall rule that
      permits your workstation and not the
      container's subnet
    - the container has no route out (Module 7,
      Chapter 3)

  TEST FROM INSIDE THE CONTAINER, not from the
  host. They are different network environments.
─────────────────────────────────────────

4. Intermittent Problems

Why They Are Hard
─────────────────────────────────────────
  You cannot reproduce on demand, so you cannot
  test a hypothesis quickly, and any change appears
  to work until it fails again.
─────────────────────────────────────────

The Approach
─────────────────────────────────────────
  1. INSTRUMENT FIRST. You cannot debug what you
     cannot observe. Continuous monitoring, and a
     capture running with a ring buffer:

       tcpdump -i eth0 -w cap-%Y%m%d-%H%M.pcap \
               -G 300 -W 12 'port 443'
       # rotate every 5 minutes, keep the last 12

  2. CORRELATE. When it happens, what ELSE happens?
     A deploy, a batch job, a backup, a peak, a
     cron?

  3. LOOK FOR PERIODICITY. Every 5 minutes? Hourly?
     Periodicity means a SCHEDULED cause.

  4. CHECK THE OBVIOUS COINCIDENCES.
       every ~60s      ── a keepalive or health
                          check interval
       every ~5 min    ── a cron job or NAT timeout
       at the same time ── a scheduled task
       under load only  ── contention (symptom 3)

  5. WIDEN THE LOGGING TEMPORARILY. Verbose logs on
     the suspected component, with timestamps
     synchronised by NTP (Module 5, Chapter 5) so
     they can actually be correlated.
─────────────────────────────────────────

The Ring Buffer Is the Key Technique
─────────────────────────────────────────
  You cannot capture continuously — the files are
  enormous.

  A rotating buffer keeps the last N minutes. When
  the problem occurs, STOP the capture, and you
  have the packets from the moment it happened.

  Without it, you notice the problem, start a
  capture, and wait for it to recur — which it does
  at 3am on a Sunday.
─────────────────────────────────────────

5. Common Wrong Turns

Mistakes That Cost Hours
─────────────────────────────────────────
  BLAMING THE NETWORK FIRST
    Most "network problems" are applications,
    configuration or DNS. Check ttfb before
    packet-capturing (Chapter 1).

  CHANGING SEVERAL THINGS AT ONCE
    Now you cannot attribute the fix, and you have
    added risk.

  MISREADING traceroute
    Intermediate loss and latency are usually ICMP
    rate limiting (Chapter 1). Only the destination
    hop matters.

  TESTING FROM THE WRONG PLACE
    Testing from your laptop when users are
    affected in another region tells you about your
    laptop's path.

  ASSUMING SYMMETRY
    The return path may differ from the forward
    path (Module 3, Chapter 4). A one-way problem
    is entirely possible.

  IGNORING RECENT CHANGES
    "Nothing changed" is almost never true. Check
    deploys, config management, certificate
    renewals, DNS edits, firewall rules.

  DISABLING SECURITY TO "TEST"
    Turning off TLS verification or the firewall
    (Module 6, Chapter 3) may make it work, and
    tells you very little while creating a real
    risk that outlives the investigation.
─────────────────────────────────────────

The Highest-Value Question
─────────────────────────────────────────
  "WHAT CHANGED?"

  Systems that worked yesterday and fail today
  almost always changed. A deploy, a certificate
  expiry, a DNS TTL lapsing, a rule edit, a
  dependency's own change, or simply DATA GROWTH
  crossing a threshold.

  Ask it before opening any tool.
─────────────────────────────────────────

6. A Checklist

BEFORE TOUCHING ANYTHING
─────────────────────────────────────────
  □ Describe the symptom precisely
  □ Narrow: who, what, where, when, how bad
  □ Ask what changed
  □ Confirm it is reproducible, and how

THE SEQUENCE
─────────────────────────────────────────
  □ L1-2: interface up? address? ARP to gateway?
  □ L3:   ping an IP. Then trace the path.
  □ DNS:  resolve, and compare with authoritative
  □ L4:   is the port open? refused or timeout?
  □ TLS:  handshake, certificate, protocol
  □ L7:   curl -v, and the timing breakdown
  □ Only then: packet capture

BEFORE DECLARING IT FIXED
─────────────────────────────────────────
  □ Do you understand WHY it broke?
    ── if not, it will recur
  □ Did you verify the fix, not just observe it
    stop?
  □ Have you reverted the changes that did not
    help?
  □ Is there monitoring that would catch it
    sooner next time?
  □ Have you written down what happened?
─────────────────────────────────────────

The Habit Worth Building
─────────────────────────────────────────
  "It works now" is not a diagnosis.

  A problem that resolved without your
  understanding why will return — usually at a
  worse moment, and you will start the
  investigation from zero.

  Spending twenty more minutes to find the actual
  cause is almost always cheaper than the second
  incident.
─────────────────────────────────────────

7. Summary & Next Steps

Key Takeaways

  • Most diagnostic work is describing the problem accurately; narrowing by who, what, where, when and how bad often identifies the cause before any tool is opened.
  • "Works for small requests, hangs on large" has essentially one cause: blocked ICMP breaking path MTU discovery.
  • Slowness only at peak means contention rather than a broken path, and intermittent resets usually mean an idle timeout somewhere in the path.
  • Change one thing at a time, write down what you tested, and never treat "it works now" as a diagnosis.

Concept Check

  1. A user reports uploads over 1 MB failing while everything else works. What is your first hypothesis and how do you confirm it?
  2. Why does "slow only at peak" rule out a large class of causes immediately?
  3. Why is a rotating packet capture the right technique for an intermittent problem?

Next Chapter

→ Chapter 3: Capstone — HTTP From Sockets

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to Computer Networks Index