Generative AI

Working With Text Generation

Context Windows & Long Inputs

The context window is the maximum number of tokens a model can attend to in one request — and it is shared between everything you send and everything it generat

JrCodex·7 min read

Jr Codex Generative AI Notes

Level: Intermediate Prerequisites: Chapter 3: Structured Generation Time to complete: ~20 minutes

Table of Contents

  1. The Context Window Is a Budget
  2. Long Context Is Not Free Context
  3. Managing Conversation History
  4. Patterns for Documents Larger Than the Window
  5. Prompt Caching
  6. Choosing a Strategy
  7. Summary & Next Steps

1. The Context Window Is a Budget

The context window is the maximum number of tokens a model can attend to in one request — and it is shared between everything you send and everything it generates.

What Competes for the Same Space
─────────────────────────────────────────
  ┌──────────────── context window ──────────────────┐
  │ system prompt │ tools │ history │ documents │ out │
  └───────────────────────────────────────────────────┘

  Reserve output space FIRST. A request that fills the
  window with input leaves nothing to answer with, and
  the call either errors or truncates.
─────────────────────────────────────────

def build_request(system, history, documents, window=128_000, reserve_output=4_000):
    budget = window - reserve_output - count_tokens(system)
    kept_docs = fit_within(documents, int(budget * 0.6))     # 60% to retrieved content
    kept_hist = fit_within(history, budget - count_tokens(kept_docs))
    return system, kept_hist, kept_docs

Explicit budgeting beats hoping. Every production system that handles variable-length input needs a function shaped roughly like this one.

2. Long Context Is Not Free Context

A 200k-token window does not mean you should use 200k tokens. Three costs rise with input length, and one of them is not obvious.

The Three Costs
─────────────────────────────────────────
  1. MONEY      linear in input tokens. A 100k-token
                prompt sent on every turn of a chat is
                a large recurring bill.

  2. LATENCY    the model must process the whole prompt
                before the FIRST output token appears.
                Long prompts feel slow even when
                streaming.

  3. ACCURACY   this is the surprising one.
─────────────────────────────────────────

"Lost in the Middle"
─────────────────────────────────────────
  Retrieval accuracy within a long prompt is not flat.
  Models attend most reliably to the BEGINNING and the
  END of the context, and least reliably to the middle.

     accuracy
       ▲
       │ ██                              ██
       │ ██ ██                      ██  ██
       │ ██ ██ ██  ██  ██  ██  ██  ██  ██
       └────────────────────────────────────► position
         start          middle           end

  Consequence: a relevant fact buried at 55% depth in a
  long document may be MISSED even though it is
  technically "in context".
─────────────────────────────────────────

Two Practical Rules
─────────────────────────────────────────
  - Put the QUESTION and the most important constraints
    at the END of the prompt, after the documents.
  - Retrieving 5 relevant chunks beats pasting 200 pages.
    A long window is a convenience, NOT a replacement
    for retrieval (NLP Notes, Module 8).
─────────────────────────────────────────

3. Managing Conversation History

Because the API is stateless (Chapter 1), an unmanaged chat grows without bound. Three strategies, in increasing sophistication.

Strategy 1 — SLIDING WINDOW
─────────────────────────────────────────
  Keep the system prompt + the last N turns.
  Cheap, simple, and forgets the user's name from
  turn 2. Fine for short task-focused sessions.

Strategy 2 — SUMMARISE AND COMPACT
─────────────────────────────────────────
  When history exceeds a threshold, replace the oldest
  turns with a generated summary of them.

  [system] [SUMMARY of turns 1-20] [turns 21-30]

  Keeps long-range facts at a fraction of the tokens.
  Cost: one extra call, and summarisation is lossy —
  each compaction can drop a detail permanently.

Strategy 3 — RETRIEVE FROM HISTORY
─────────────────────────────────────────
  Store every turn in a vector store; retrieve only the
  turns relevant to the CURRENT message.

  Scales to unbounded history at constant prompt size.
  This is the direct precursor to agent memory —
  Agentic AI Notes, Module 3.
─────────────────────────────────────────

def compact(client, messages, max_tokens=8_000, keep_recent=6):
    if count_messages(messages) <= max_tokens:
        return messages
 
    system, body = messages[0], messages[1:]
    old, recent = body[:-keep_recent], body[-keep_recent:]
 
    summary = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user",
                   "content": "Summarise this conversation. Preserve every decision, "
                              "name, number and open question.\n\n" + render(old)}],
    ).choices[0].message.content
 
    return [system,
            {"role": "user", "content": f"[Earlier conversation summary]\n{summary}"},
            *recent]

The instruction to preserve decisions, names, numbers and open questions is doing real work. A generic "summarise this" will produce a readable paragraph that has quietly dropped the one figure the next turn needs.

4. Patterns for Documents Larger Than the Window

MAP-REDUCE
─────────────────────────────────────────
  Split ──► process each chunk independently ──►
  combine the results.

  Chunks are processed IN PARALLEL, so this is fast.
  Each chunk is blind to the others, so it is wrong for
  anything needing cross-chunk reasoning.

  Good for: per-section summaries, extracting all
  entities, classifying every clause.

REFINE (sequential)
─────────────────────────────────────────
  answer = process(chunk_1)
  for each later chunk:
      answer = revise(answer, chunk)

  Each step sees the accumulated answer, so context
  carries forward. Strictly serial and therefore slow;
  early chunks influence the result disproportionately.

  Good for: a single coherent summary of a long report.

RETRIEVE-THEN-GENERATE
─────────────────────────────────────────
  Index the document, retrieve only the chunks relevant
  to the question, generate from those.

  Cheapest and usually most accurate for QUESTION
  ANSWERING. Cannot answer questions requiring the
  whole document ("how many times does X appear?").

  → NLP Notes, Module 8, Chapter 2
─────────────────────────────────────────

Picking Between Them
─────────────────────────────────────────
  "What does the document say about X?"  ──► retrieve
  "Summarise the whole document"         ──► refine
  "Do this to every section"             ──► map-reduce
─────────────────────────────────────────

5. Prompt Caching

When many requests share a long identical prefix — a big system prompt, a fixed document, a tool list — providers can cache the processed prefix.

What Caching Changes
─────────────────────────────────────────
  Request 1:  [ 40k-token manual ][ question A ]
                └── processed and CACHED ──┘

  Request 2:  [ 40k-token manual ][ question B ]
                └── cache HIT: cheap and fast ──┘

  Typical effect: large discount on cached input tokens
  and a substantial drop in time-to-first-token.
─────────────────────────────────────────

The One Rule That Makes Caching Work
─────────────────────────────────────────
  Caching matches on an EXACT PREFIX.

  So: put everything STATIC at the FRONT, and
  everything VARIABLE at the BACK.

  ✗  [timestamp][user name][manual][question]
       ── one changed token at the front invalidates
          the entire 40k-token cache entry

  ✓  [manual][tools][timestamp][user name][question]
─────────────────────────────────────────

Note the tension with Section 2's advice to put the question last — they agree. Static content forward, question last, is the same ordering for both reasons.

6. Choosing a Strategy

Decision Guide
─────────────────────────────────────────
  Input fits comfortably in the window
      ──► send it directly; cache the static part

  Chat that outgrows the window
      ──► sliding window (short tasks)
          summarise-and-compact (assistants)
          retrieval (long-lived agents)

  One document larger than the window
      ──► retrieve for Q&A
          refine for a whole-document summary
          map-reduce for per-section work

  Many documents
      ──► retrieval, always
─────────────────────────────────────────

7. Summary & Next Steps

Key Takeaways

  • The context window is a shared budget across system prompt, tools, history, documents and output — reserve output space explicitly before filling it.
  • Long context costs money, latency, and accuracy: attention is weakest in the middle, so a large window is not a substitute for retrieval.
  • Manage history with a sliding window, summarise-and-compact, or retrieval, in ascending order of history length and cost.
  • Prompt caching matches on exact prefixes, so ordering the prompt static-first and variable-last is what makes it effective.

Concept Check

  1. Why can a fact that is genuinely inside a 150k-token prompt still be missed by the model?
  2. A summariser is compacting chat history but the assistant keeps forgetting the user's stated budget. What is wrong, and what is the fix?
  3. Your prompt begins with the current timestamp followed by a 30k-token policy document. Why is the cache hit rate near zero, and what is the one-line change?

Module 2 Complete — What's Next

You can now drive a text model as a reliable software component: shaped output, controlled sampling, managed context. Module 3 moves to a modality where the controls are entirely different — image generation — and picks up the latent-space thread from Module 1.

Next Module

→ Module 3: Image Generation

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to Generative AI Index