Generative AI

Working With Text Generation

Controlling the Output

NLP Module 5, Chapter 1 introduced the decoding strategies. This chapter is about operating them: what each dial actually does to the distribution, how they int

JrCodex·7 min read

Jr Codex Generative AI Notes

Level: Beginner–Intermediate Prerequisites: Chapter 1: Calling a Generative Model; NLP & LLM Notes, Module 5, Chapter 1 Time to complete: ~20 minutes

Table of Contents

  1. The Control Panel
  2. Temperature — What It Really Does
  3. Top-p and Top-k — Trimming the Tail
  4. Penalties and Logit Bias
  5. Stop Sequences and Length
  6. The Determinism Problem
  7. Settings by Task
  8. Summary & Next Steps

1. The Control Panel

NLP Module 5, Chapter 1 introduced the decoding strategies. This chapter is about operating them: what each dial actually does to the distribution, how they interact, and what to set them to.

Where Every Dial Acts
─────────────────────────────────────────
  model produces LOGITS (one raw score per token)
        │
        ├── logit_bias        adds a fixed nudge per token
        ├── frequency_penalty subtracts by how OFTEN a token
        ├── presence_penalty  subtracts if a token appeared
        │
        ▼
  divide by TEMPERATURE, then softmax ──► probabilities
        │
        ├── top_k    keep the k highest
        ├── top_p    keep the smallest set summing to p
        │
        ▼
  renormalise and SAMPLE one token
        │
        ├── stop sequences   end generation early
        └── max_tokens       hard cap
─────────────────────────────────────────

Everything above happens per token, at every step. That ordering matters: penalties apply before temperature, and top-p applies after it.

2. Temperature — What It Really Does

Temperature divides the logits before the softmax. It does not add randomness — it changes how sharp the existing distribution is.

The Same Logits at Three Temperatures
─────────────────────────────────────────
  Raw scores:   "sunny" 4.0   "cold" 3.0   "purple" 1.0

  T = 0.2   ──►  0.99   0.01   0.00    near-greedy
  T = 1.0   ──►  0.71   0.26   0.03    the model's own view
  T = 1.8   ──►  0.48   0.34   0.18    flattened; odd tokens
                                       become reachable
─────────────────────────────────────────

The Two Things People Get Wrong
─────────────────────────────────────────
  1. T = 0 is NOT "no randomness" in a distributed
     system — it is greedy decoding, which is still
     not bit-reproducible. See Section 6.

  2. HIGH temperature does not make the model more
     creative. It makes it more willing to pick tokens
     it rated as UNLIKELY. Above about 1.2 that mostly
     means incoherence, not originality.
─────────────────────────────────────────

The useful working range for almost all applications is 0.0 to 1.0. Reach above 1.0 only for deliberate brainstorming, and expect to discard more output.

3. Top-p and Top-k — Trimming the Tail

Temperature reshapes the whole distribution. Top-p and top-k instead truncate it, deleting the long tail of implausible tokens before sampling.

Top-k vs Top-p on the Same Step
─────────────────────────────────────────
  Sorted probabilities:
    0.55  0.20  0.12  0.06  0.04  0.02  0.01 ...

  top_k = 3    keeps the first 3, ALWAYS 3, regardless
               of whether the model was confident

  top_p = 0.9  keeps 0.55 + 0.20 + 0.12 = 0.87, then
               0.06 ──► 0.93 ≥ 0.9, so it keeps 4.
               On a CONFIDENT step (0.95 top token) it
               would keep only 1.
─────────────────────────────────────────

Top-p adapts, top-k does not — which is why top-p (nucleus sampling) is the default in most APIs and the one to reach for. A fixed k is too permissive on confident steps and too restrictive on genuinely ambiguous ones.

Do Not Tune Both at Once
─────────────────────────────────────────
  Temperature and top_p both control the same thing —
  the effective breadth of the sample. Changing both
  makes the effect of each impossible to reason about.

  Pick ONE as your dial. Convention: hold top_p at 1.0
  and tune temperature, or hold temperature at 1.0 and
  tune top_p. Most teams tune temperature.
─────────────────────────────────────────

4. Penalties and Logit Bias

Two penalties address repetition, and they differ in a way worth knowing.

ParameterEffectUse when
frequency_penaltyReduces a token's score in proportion to how many times it has already appearedOutput loops or over-uses a phrase
presence_penaltyReduces a token's score by a flat amount if it appeared at allYou want the model to move on to new topics
logit_biasAdds a fixed value to specific token IDs, every stepYou must ban or force a specific token
# Ban a token outright: -100 makes it effectively unreachable.
enc = tiktoken.encoding_for_model("gpt-4o-mini")
banned = {enc.encode(" maybe")[0]: -100, enc.encode(" perhaps")[0]: -100}
 
response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=messages,
    temperature=0.7,
    frequency_penalty=0.4,      # discourage repeated phrasing
    logit_bias=banned,
)
Penalties Are a Blunt Instrument
─────────────────────────────────────────
  A frequency penalty cannot tell the difference between
  a repeated FILLER phrase and a repeated TECHNICAL TERM
  that the answer genuinely requires.

  Set above about 0.5 and a paragraph about "the encoder"
  will start avoiding the word "encoder". Prefer fixing
  repetition in the PROMPT; use penalties as a last resort.
─────────────────────────────────────────

5. Stop Sequences and Length

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "List three fruits, one per line."}],
    stop=["\n\n", "4."],       # cut generation the moment either appears
    max_tokens=100,
)
Two Different Kinds of Ending
─────────────────────────────────────────
  stop sequence   a CLEAN, semantic ending. The stop
                  text itself is NOT included in the
                  output. finish_reason = "stop".

  max_tokens      a HARD cut, possibly mid-word.
                  finish_reason = "length".

  Use stop sequences to define structure. Use max_tokens
  ONLY as a cost and runaway ceiling — never as your
  intended way to end a response.
─────────────────────────────────────────

6. The Determinism Problem

A common and costly assumption: that temperature=0 makes a call reproducible.

Why temperature=0 Is Not Reproducible
─────────────────────────────────────────
  Greedy decoding picks the ARGMAX token. But the
  logits themselves vary slightly between runs because:

  - GPU floating-point reduction order is not fixed
  - batching differs: your request is processed
    alongside different neighbours each time
  - providers update model weights behind a stable
    model name

  When two tokens are nearly tied, a 1e-7 difference
  flips the argmax — and the two continuations then
  diverge completely.
─────────────────────────────────────────

# `seed` improves reproducibility but does NOT guarantee it.
response = client.chat.completions.create(
    model="gpt-4o-mini", messages=messages, temperature=0, seed=42,
)
print(response.system_fingerprint)      # CHANGES when the backend changes —
                                        # different fingerprint means results
                                        # are not comparable to earlier runs
Design for Variance, Not Against It
─────────────────────────────────────────
  - Never write tests that assert on exact model output.
    Assert on SCHEMA, on invariants, on score thresholds.
  - Cache by prompt hash if you need the same answer
    twice (Chapter 4).
  - Pin explicit model VERSIONS, not floating aliases.
  - Log system_fingerprint alongside evaluation results.
─────────────────────────────────────────

7. Settings by Task

A starting point, to be adjusted with real evaluation (Module 5) rather than by feel.

Recommended Starting Values
─────────────────────────────────────────
  TASK                      temp   top_p  freq_pen
  ─────────────────────────────────────────
  Extraction / JSON          0.0    1.0     0.0
  Classification             0.0    1.0     0.0
  Factual Q&A over docs      0.1    1.0     0.0
  Summarisation              0.3    1.0     0.1
  Code generation            0.2    1.0     0.0
  Conversational assistant   0.7    1.0     0.2
  Marketing copy             0.8    1.0     0.3
  Brainstorming variants     1.0    1.0     0.5
─────────────────────────────────────────

The pattern: the more the output is checked against something external, the lower the temperature. Anything with a schema, a source document, or a compiler goes near zero. Anything judged by human taste goes higher.

8. Summary & Next Steps

Key Takeaways

  • Temperature sharpens or flattens the existing distribution; top-p and top-k truncate its tail. Tune one, not both.
  • Top-p adapts its candidate pool to the model's confidence at each step, which is why it is preferred over a fixed top-k.
  • Penalties are blunt and will suppress technical terms the answer needs — fix repetition in the prompt first.
  • temperature=0 gives greedy decoding, not reproducibility. Test against schemas and thresholds, never against exact strings.

Concept Check

  1. Why does raising temperature to 1.8 usually produce incoherence rather than creativity?
  2. Give a concrete step where top-p keeps one token and top-k=5 keeps five. Which behaviour is preferable, and why?
  3. A teammate's test asserts the model returns the exact string "Approved." It passes locally and fails in CI. Explain the mechanism and propose a better assertion.

Next Chapter

→ Chapter 3: Structured Generation

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to Generative AI Index