Computer Networks

Practice And Capstone

The Diagnostic Toolkit

1-2 is the interface up? ip link, ethtool

JrCodex·9 min read

Jr Codex Computer Networks Notes

Level: Advanced Prerequisites: Module 7, Chapter 4: Service-to-Service Communication Time to complete: ~20 minutes

Table of Contents

  1. The Tools by Layer
  2. Layer 1-2 — Is the Link Up
  3. Layer 3 — Reachability and Path
  4. DNS
  5. Layer 4 — Connections and Ports
  6. Layer 7 — What the Application Sees
  7. Packet Capture
  8. Summary & Next Steps

1. The Tools by Layer

The Map
─────────────────────────────────────────
  LAYER  QUESTION                  TOOL
  ─────────────────────────────────────────
   1-2   is the interface up?      ip link, ethtool
   2     who is on my LAN?         ip neigh, arp
   3     can I reach that IP?      ping
   3     what path does it take?   traceroute, mtr
   -     does the name resolve?    dig, nslookup
   4     is the port open?         nc, telnet
   4     what connections exist?   ss, netstat
   7     what does the app see?    curl, openssl
   ALL   what is actually on the   tcpdump,
         wire?                     Wireshark
─────────────────────────────────────────

Use It In Order
─────────────────────────────────────────
  Module 1, Chapter 2's bottom-up rule. The FIRST
  layer that fails localises the problem, and
  everything above it is irrelevant until that is
  fixed.

  Testing top-down means guessing.
─────────────────────────────────────────

2. Layer 1-2 — Is the Link Up

ip link show                    # is the interface UP, and does it have a carrier?
# 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP
#                                    └ LOWER_UP = a cable is connected
 
ip addr show                    # do I have an address at all?
# inet 192.168.1.10/24 ...
# inet 169.254.x.x     ← DHCP FAILED (Module 3, Chapter 1)
 
ethtool eth0                    # negotiated speed and duplex
ethtool -S eth0 | grep -i err   # interface error counters
 
ip neigh show                   # the ARP cache (Module 2, Chapter 4)
# 192.168.1.1 dev eth0 lladdr 00:1a:2b:3c:4d:5e REACHABLE
What These Rule Out Fast
─────────────────────────────────────────
  NO LOWER_UP          a physical problem. Cable,
                       port, or the switch.

  169.254.x.x          DHCP did not answer. Stop
                       looking at routing and DNS
                       entirely.

  RISING ERROR COUNTS  a bad cable, a duplex
                       mismatch, or interference.
                       Symptom: intermittent slowness
                       with no obvious cause.

  NO ARP ENTRY FOR     you cannot reach your own
  THE GATEWAY          gateway. Nothing above layer
                       2 will work.
─────────────────────────────────────────

3. Layer 3 — Reachability and Path

ping -c 20 8.8.8.8
# 20 packets transmitted, 20 received, 0% packet loss
# rtt min/avg/max/mdev = 12.1/14.3/38.9/4.2 ms
#     └ propagation  └ typical  └ a queuing spike  └ JITTER
 
# Ping an IP FIRST, then a name. It separates connectivity from DNS.
ping -c 3 8.8.8.8          # works, but the next fails ──► DNS problem
ping -c 3 example.com
 
ip route get 8.8.8.8       # which route and interface WOULD be used?
mtr --report --report-cycles 50 example.com     # better than traceroute
# HOST                    Loss%  Snt  Last  Avg  Best  Wrst StDev
# 1. 192.168.1.1           0.0%   50   0.8   0.9   0.7   1.4   0.1
# 2. isp-gw.example.net    0.0%   50  12.3  12.8  11.9  18.2   1.1
# 3. core-r1.example.net  38.0%   50  13.1  13.4  12.8  15.9   0.7   ← ?
# 4. core-r2.example.net   0.0%   50  13.5  13.9  13.1  16.4   0.8
# 5. example.com           0.0%   50  14.2  14.6  13.8  19.1   1.2
READING LOSS AT AN INTERMEDIATE HOP
─────────────────────────────────────────
  Hop 3 shows 38% loss. Hops 4 and 5 show none.

  THIS IS NOT A PROBLEM.

  That router is rate-limiting the ICMP replies it
  generates for you (Module 3, Chapter 5) while
  forwarding your traffic perfectly. If loss did
  not continue to the final hop, the path is fine.

  ONLY LOSS THAT PERSISTS TO THE DESTINATION
  MATTERS. Similarly, a latency jump matters only
  if every subsequent hop inherits it.

  Misreading this sends people chasing a
  non-existent fault in someone else's network.
─────────────────────────────────────────

4. DNS

dig example.com A +short                # just the answer
dig example.com A                       # full: flags, TTL, which server answered
dig +trace example.com                  # the full delegation walk (Module 5, Ch.1)
dig @8.8.8.8 example.com                # bypass your local resolver
dig @ns1.example.com example.com        # ask the AUTHORITATIVE server directly
dig example.com MX +short
dig -x 93.184.216.34 +short             # reverse
The Comparison That Diagnoses Everything
─────────────────────────────────────────
  Ask your resolver, then ask the authoritative
  server:

    dig example.com                     # cached
    dig @ns1.example.com example.com    # the truth

  DIFFERENT ANSWERS ──► a STALE CACHE. Wait for the
  TTL, or flush.

  SAME ANSWER, AND WRONG ──► the RECORD is wrong.
  Fix the DNS, not the cache.

  NO ANSWER FROM AUTHORITATIVE ──► the record does
  not exist, or the delegation is broken.

  This one comparison separates the three DNS
  failure modes in ten seconds.
─────────────────────────────────────────

5. Layer 4 — Connections and Ports

nc -zv example.com 443              # is the port open? -z = scan, no data
nc -zv example.com 22
# Connection to example.com 443 port [tcp/https] succeeded!
 
# Test with a timeout, so a filtered port fails fast rather than hanging.
timeout 3 nc -zv example.com 443 || echo "unreachable or filtered"
ss -tlnp                    # what is LISTENING here?
# State  Recv-Q Send-Q  Local Address:Port
# LISTEN 0      128           0.0.0.0:22
# LISTEN 0      511         127.0.0.1:8080     ← LOCALHOST ONLY
 
ss -tan state established | wc -l           # how many connections?
ss -tan | awk '{print $1}' | sort | uniq -c # by state (Module 4, Chapter 1)
ss -tin dst 10.0.1.5                        # per-connection RTT, cwnd, retransmits
THREE FINDINGS THAT SOLVE THE PROBLEM
─────────────────────────────────────────
  LISTENING ON 127.0.0.1 ONLY
    The service works locally and is unreachable
    from anywhere else. Bind to 0.0.0.0.
    ── an extremely common cause of "the container
       cannot reach my service"

  CONNECTION REFUSED vs TIMEOUT
    REFUSED ──► the packet ARRIVED and a TCP RST
                came back. Nothing is listening.
                Routing and firewalls are FINE.
    TIMEOUT ──► nothing came back at all. A
                firewall is dropping it, or routing
                is wrong.
    Two completely different investigations.

  MANY CLOSE_WAIT
    An application not closing sockets (Module 4,
    Chapter 1). Always a code bug.
─────────────────────────────────────────

6. Layer 7 — What the Application Sees

curl -v https://example.com                 # the whole exchange, including TLS
 
# The timing breakdown — Module 1, Chapter 5's most useful command.
curl -w "dns:%{time_namelookup}s connect:%{time_connect}s tls:%{time_appconnect}s \
ttfb:%{time_starttransfer}s total:%{time_total}s proto:%{http_version} code:%{http_code}\n" \
     -o /dev/null -s https://example.com
 
curl -I https://example.com                 # headers only
curl --resolve example.com:443:203.0.113.5 https://example.com    # bypass DNS
curl --http1.1 / --http2 / --http3 ...      # force a protocol version
# TLS specifically (Module 6, Chapter 3)
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>&1 \
  | grep -E "Protocol|Cipher|Verify return code"
 
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -dates -ext subjectAltName
Reading the curl Breakdown
─────────────────────────────────────────
  dns HIGH        resolver problem (Section 4)
  connect HIGH    propagation delay, or a slow
                  handshake — the server is far or
                  the path is bad
  tls HIGH        extra handshake round trips, or
                  an OCSP lookup stalling
  ttfb HIGH       THE SERVER IS SLOW. The network
                  is fine.
  total >> ttfb   bandwidth, or a large response

  This single command separates "the network is
  slow" from "the server is slow" — which are
  investigated completely differently, and are
  confused constantly.
─────────────────────────────────────────

7. Packet Capture

The tool of last resort, and the only one that shows you the truth.

sudo tcpdump -i eth0 -nn 'host 93.184.216.34 and port 443'
#              │    │   └ the filter
#              │    └ -nn: do not resolve names or ports (faster, clearer)
#              └ interface
 
# Just the handshake — SYN, SYN-ACK, RST (Module 4, Chapter 3)
sudo tcpdump -i any -nn 'tcp[tcpflags] & (tcp-syn|tcp-rst) != 0'
 
# Capture to a file and analyse in Wireshark.
sudo tcpdump -i eth0 -w capture.pcap -s 0 'port 443'
 
# See DNS queries and answers in full.
sudo tcpdump -i any -nn -s 0 'port 53'
Reading a Capture
─────────────────────────────────────────
  SYN with no SYN-ACK
    Nothing is listening, or a firewall silently
    dropped it. Compare with an RST, which means a
    definite refusal.

  RETRANSMISSIONS
    The same sequence number sent repeatedly. Loss
    (Module 4, Chapter 3).

  ZERO WINDOW
    The receiver's buffer is full. The APPLICATION
    is not reading fast enough — not a network
    problem (Module 4, Chapter 4).

  RST MID-CONNECTION
    Something aborted it. An application crash, a
    firewall, or an idle timeout.

  DUPLICATE ACKS
    Out-of-order delivery or loss; fast retransmit
    is about to happen.
─────────────────────────────────────────

Wireshark, Practically
─────────────────────────────────────────
  Open the pcap, then:

    Statistics ──► Conversations
      who is talking to whom, and how much

    Analyze ──► Expert Information
      Wireshark's OWN list of anomalies:
      retransmissions, resets, malformed packets.
      START HERE — it does the first pass for you.

    Follow ──► TCP Stream
      reassembles the whole conversation as text

    Filter: tcp.analysis.flags
      show only segments Wireshark flagged as
      problematic
─────────────────────────────────────────

When to Reach for a Capture
─────────────────────────────────────────
  ONLY when the higher-level tools have given
  contradictory or insufficient answers.

  A capture on a busy interface produces enormous
  files and takes real time to read. ALWAYS FILTER
  at capture time, by host and port.

  And remember it contains real data — treat
  captures as sensitive.
─────────────────────────────────────────

8. Summary & Next Steps

Key Takeaways

  • Work bottom-up: the first layer that fails localises the problem and makes everything above it irrelevant until fixed.
  • Loss shown at an intermediate mtr hop is usually ICMP rate limiting, not a fault — only loss that persists to the destination matters.
  • Comparing your resolver's answer with the authoritative server's separates a stale cache, a wrong record and a broken delegation in seconds.
  • Connection refused and connection timeout are completely different findings: refused proves the packet arrived, timeout means something dropped it.

Concept Check

  1. mtr shows 40% loss at hop 4 and 0% at hops 5 through 9. What does this mean?
  2. What is the practical difference between "connection refused" and "connection timed out" when diagnosing?
  3. Your curl breakdown shows a fast connect and a slow time-to-first-byte. Where is the problem, and where is it not?

Next Chapter

→ Chapter 2: Debugging Methodically

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to Computer Networks Index