Computer Networks

Modern And Distributed Networking

Service-to-Service Communication

cannot fail can time out, be refused,

JrCodex·10 min read

Jr Codex Computer Networks Notes

Level: Advanced Prerequisites: Chapter 3: Cloud Networking Time to complete: ~20 minutes

Table of Contents

  1. The Network Becomes the Bottleneck
  2. gRPC
  3. Timeouts, Retries and Backoff
  4. Circuit Breakers
  5. Service Discovery and the Mesh
  6. Distributed Tracing
  7. Summary & Next Steps

1. The Network Becomes the Bottleneck

What Changes With Microservices
─────────────────────────────────────────
  A function call becomes a NETWORK CALL.

    IN-PROCESS         NETWORK
    ~1 nanosecond      ~1 millisecond (same zone)
    cannot fail        can time out, be refused,
                       or silently vanish
    no serialisation   encode, transmit, decode
    one failure mode   many
─────────────────────────────────────────

THE FALLACIES OF DISTRIBUTED COMPUTING
─────────────────────────────────────────
  Assumptions that are false, and that every
  distributed system eventually violates:

    1. the network is reliable
    2. latency is zero
    3. bandwidth is infinite
    4. the network is secure
    5. topology does not change
    6. there is one administrator
    7. transport cost is zero
    8. the network is homogeneous

  Every module of this curriculum has demonstrated
  one of these to be false. This chapter is about
  designing as though you believe that.
─────────────────────────────────────────

The Latency Arithmetic
─────────────────────────────────────────
  One user request fanning out to 10 services, each
  1 ms:

    SEQUENTIAL  10 ms, plus 10 failure
                opportunities
    PARALLEL    ~1 ms, but bounded by the SLOWEST

  And with 10 services each 99.9% available, the
  end-to-end availability of a sequential chain is
  0.999^10 ≈ 99.0%.

  Ten nines-and-a-bit services compose into
  something noticeably worse. Composition
  MULTIPLIES failure probability, which is the
  central difficulty.
─────────────────────────────────────────

2. gRPC

What It Is
─────────────────────────────────────────
  RPC over HTTP/2 (Module 5, Chapter 3) with
  Protocol Buffers for serialisation.

  You define the service in a .proto file, and
  generate typed client and server code for any
  language.
─────────────────────────────────────────

syntax = "proto3";
package orders;
 
service OrderService {
  rpc GetOrder    (GetOrderRequest) returns (Order);
  rpc ListOrders  (ListOrdersRequest) returns (stream Order);   // server streaming
  rpc WatchOrders (stream WatchRequest) returns (stream Order); // bidirectional
}
 
message GetOrderRequest { string order_id = 1; }
 
message Order {
  string id       = 1;
  string customer = 2;
  int64  total_cents = 3;
  Status status   = 4;
  enum Status { PENDING = 0; PAID = 1; SHIPPED = 2; }
}
Why It Suits Service-to-Service
─────────────────────────────────────────
  BINARY AND COMPACT   protobuf is far smaller than
                       JSON, and faster to parse

  SCHEMA-FIRST         the contract is explicit,
                       versioned and machine-checked
                       — no arguing about response
                       shapes

  CODE GENERATION      typed clients in every
                       language; a field rename
                       breaks at COMPILE time, not
                       in production

  STREAMING            four modes: unary, server
                       stream, client stream,
                       bidirectional
                       ── over HTTP/2's multiplexed
                          streams

  HTTP/2 MULTIPLEXING  many concurrent calls on one
                       connection, avoiding
                       connection churn
─────────────────────────────────────────

Where REST Is Still Better
─────────────────────────────────────────
  ✗ BROWSER CLIENTS — gRPC needs grpc-web and a
    proxy; the browser cannot speak raw gRPC
  ✗ PUBLIC APIs — external developers expect JSON
    over HTTP and existing tooling
  ✗ DEBUGGING — you cannot curl a binary protocol
    or read it in a proxy log
  ✗ SIMPLE CRUD — the schema and codegen machinery
    is not worth it

  THE USUAL SPLIT: REST or GraphQL at the edge for
  clients; gRPC between internal services.
─────────────────────────────────────────

3. Timeouts, Retries and Backoff

THE MOST IMPORTANT RULE
─────────────────────────────────────────
  EVERY network call needs a TIMEOUT.

  A call without one waits forever. Under load,
  every thread or connection ends up blocked on the
  same slow dependency, and a service that is
  merely SLOW takes down everything calling it.

  This is the single most common cause of
  cascading failure in distributed systems.
─────────────────────────────────────────

Setting Timeouts Sensibly
─────────────────────────────────────────
  Base them on the p99 latency, not the average.
  Roughly 2-3× p99.

  BUDGET them across a call chain. If the user's
  request has 1,000 ms, and A calls B calls C:

    A's budget  1000 ms
    B's budget   700 ms   (A keeps time to respond)
    C's budget   400 ms

  Pass the REMAINING budget as a deadline in the
  request. gRPC does this natively.

  Without budgeting, inner calls happily consume
  more time than the outer caller will ever wait
  for — work that is discarded the moment it
  completes.
─────────────────────────────────────────

import random, time
 
def retry(fn, attempts=3, base=0.05, cap=2.0, retryable=(TimeoutError, ConnectionError)):
    """Exponential backoff with FULL JITTER."""
    for i in range(attempts):
        try:
            return fn()
        except retryable:
            if i == attempts - 1:
                raise
            backoff = min(cap, base * (2 ** i))
            time.sleep(random.uniform(0, backoff))     # ← FULL jitter, not backoff/2
RETRY ONLY WHAT IS SAFE
─────────────────────────────────────────
  RETRY        timeouts, connection failures, 502,
               503, 504 — and only for IDEMPOTENT
               operations (Module 5, Chapter 2)

  DO NOT RETRY 400, 401, 403, 404, 422 — they will
               fail identically forever, and you
               burn budget and latency

  For non-idempotent operations, use an IDEMPOTENCY
  KEY so the retry is safe (DBMS Notes, Module 7,
  Chapter 5).
─────────────────────────────────────────

WHY JITTER, SPECIFICALLY
─────────────────────────────────────────
  Without it, every client that failed at the same
  moment retries at the SAME moment.

  The service recovers, is immediately hit by a
  synchronised wave, fails again, and the cycle
  repeats. A THUNDERING HERD that prevents recovery.

  Random jitter spreads the retries. It is one line
  of code and it is the difference between a
  service recovering and a service oscillating.

  It is the same insight as Ethernet's backoff
  (Module 2, Chapter 3).
─────────────────────────────────────────

RETRY AMPLIFICATION
─────────────────────────────────────────
  Retries at EVERY layer multiply.

    3 retries at the gateway
    × 3 at service A
    × 3 at service B
    = 27 requests reaching the struggling service C

  Your retry logic is now a denial of service
  attack on your own infrastructure.

  RULE: retry at ONE layer, usually the outermost.
  Or use a retry BUDGET — cap retries at a
  percentage of total requests, so they cannot
  dominate under widespread failure.
─────────────────────────────────────────

4. Circuit Breakers

The Idea
─────────────────────────────────────────
  When a dependency is failing, STOP CALLING IT.

  Failing fast is better than waiting for timeouts
  — it frees your threads, and it gives the
  struggling service room to recover.
─────────────────────────────────────────

The Three States
─────────────────────────────────────────
  CLOSED     normal. Requests pass. Failures are
             counted.
      │ failure rate exceeds the threshold
      ▼
  OPEN       requests FAIL IMMEDIATELY without
             being attempted.
      │ after a cooldown
      ▼
  HALF-OPEN  allow a FEW trial requests.
             succeed ──► CLOSED
             fail    ──► OPEN again
─────────────────────────────────────────

import time
 
class CircuitBreaker:
    def __init__(self, threshold=0.5, window=20, cooldown=30, trial=3):
        self.threshold, self.window = threshold, window
        self.cooldown, self.trial = cooldown, trial
        self.results, self.state, self.opened_at, self.trials = [], "closed", None, 0
 
    def allow(self):
        if self.state == "closed":
            return True
        if self.state == "open":
            if time.time() - self.opened_at >= self.cooldown:
                self.state, self.trials = "half_open", 0     # try a few
                return True
            return False                                      # FAIL FAST
        return self.trials < self.trial
 
    def record(self, ok):
        if self.state == "half_open":
            self.trials += 1
            if not ok:
                self.state, self.opened_at = "open", time.time()
            elif self.trials >= self.trial:
                self.state, self.results = "closed", []
            return
 
        self.results.append(ok)
        self.results = self.results[-self.window:]
        if len(self.results) >= self.window:
            failure_rate = 1 - sum(self.results) / len(self.results)
            if failure_rate >= self.threshold:
                self.state, self.opened_at = "open", time.time()
Always Have a Fallback
─────────────────────────────────────────
  An open circuit means the call fails
  IMMEDIATELY. What then?

    - serve STALE cached data
    - return a degraded response — hide the
      recommendations, show the page
    - queue the work for later
    - return a clear error for this feature only

  A circuit breaker without a fallback converts a
  slow failure into a fast failure. Useful, and not
  the point. The point is to keep serving what you
  still can — Module 6's degrade rather than
  collapse.
─────────────────────────────────────────

5. Service Discovery and the Mesh

The Problem
─────────────────────────────────────────
  Instances are ephemeral. They autoscale, get
  replaced, change addresses. Hardcoding an IP is
  not an option (Chapter 3).

  SERVICE DISCOVERY answers "where is service B
  right now?"

  Two approaches:
    CLIENT-SIDE   the client queries a registry and
                  picks an instance itself
    SERVER-SIDE   the client calls a stable name;
                  a load balancer resolves it
                  (Chapter 1)
─────────────────────────────────────────

THE SERVICE MESH
─────────────────────────────────────────
  Every service instance gets a SIDECAR PROXY.
  All traffic goes through it.

    service A ──► sidecar ══► sidecar ──► service B
                    └── the mesh data plane ──┘

  The sidecars handle, uniformly and outside
  application code:
    - service discovery and load balancing
    - retries, timeouts, circuit breaking
      (Sections 3-4)
    - mTLS between services (Module 6, Chapter 3)
    - metrics, logs and traces
    - traffic splitting for canary deploys
─────────────────────────────────────────

The Honest Trade
─────────────────────────────────────────
  GAIN   consistent behaviour across every
         language, with no library to update in
         twelve codebases; mTLS everywhere without
         touching application code

  COST   an extra network hop and process per
         instance (latency and memory)
         a substantial new system to operate and
         debug
         another failure mode

  WHEN IT IS WORTH IT
    Many services, several languages, and a real
    need for uniform policy and mTLS.

  WHEN IT IS NOT
    Under roughly ten services, or one language.
    A shared library is simpler and adequate.

  A service mesh is infrastructure for a problem
  you should confirm you have.
─────────────────────────────────────────

6. Distributed Tracing

The Problem It Solves
─────────────────────────────────────────
  A user request touches twelve services. It is
  slow. WHERE?

  Each service's logs show only its own part, with
  no way to connect them. Without tracing, you are
  correlating timestamps by hand across twelve log
  streams.
─────────────────────────────────────────

The Mechanism
─────────────────────────────────────────
  TRACE ID    one id for the whole request,
              generated at the edge
  SPAN ID     one per operation
  PARENT      which span this one is nested inside

  The ids are PROPAGATED in headers across every
  hop:

    traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736
                 -00f067aa0ba902b7-01
                    └ trace id ──┘ └ span id ┘

  A backend reconstructs the tree and shows exactly
  where the time went.
─────────────────────────────────────────

from opentelemetry import trace
from opentelemetry.propagate import inject, extract
 
tracer = trace.get_tracer(__name__)
 
def handle_request(incoming_headers):
    ctx = extract(incoming_headers)                    # CONTINUE the trace
    with tracer.start_as_current_span("handle_order", context=ctx) as span:
        span.set_attribute("order.id", order_id)
 
        outgoing = {}
        inject(outgoing)                               # PROPAGATE to the next service
        response = http_client.get(inventory_url, headers=outgoing)
 
        return response
What a Trace Tells You Immediately
─────────────────────────────────────────
  ──────────────── request 840 ms ────────────────
  gateway        ▇▇                        20 ms
   auth          ▇                         10 ms
   order-svc     ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇     790 ms
    inventory    ▇▇                        15 ms
    pricing      ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇        750 ms  ◄──
     currency-api ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇        740 ms  ◄──
   notify        ▇                          5 ms

  The problem is an external currency API called
  from pricing. Ninety seconds of reading, not a
  day of correlating logs.

  IF YOU BUILD ONE THING FOR A DISTRIBUTED SYSTEM,
  BUILD TRACING. Everything else is guesswork
  without it.
─────────────────────────────────────────

7. Summary & Next Steps

Key Takeaways

  • Composing services multiplies failure probability, so ten 99.9% services in a chain give roughly 99% end-to-end availability.
  • Every network call needs a timeout, budgeted across the chain and passed as a deadline — a missing timeout is the most common cause of cascading failure.
  • Retries need jitter to avoid synchronised waves, and must happen at one layer only, or they multiply into a self-inflicted denial of service.
  • A circuit breaker without a fallback only converts slow failure into fast failure; the value is in continuing to serve what you still can.

Module 7 Complete — What's Next

You now understand the infrastructure between a user and an application. Module 8 turns all seven modules into a practical skill: diagnosing a real network problem methodically, and building something end to end.

Concept Check

  1. Why does ten sequential 99.9%-available services give substantially worse end-to-end availability?
  2. Explain retry amplification, and give two ways to prevent it.
  3. Why is a service mesh not automatically the right choice for a system with six services?

Next Module

→ Module 8: Practice and Capstone

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to Computer Networks Index