Working With Text Generation
Structured Generation
Module 1, Chapter 4 argued that reliable generative features are the ones with an automatic check on the output. Structured generation is that check, for every
JrCodex·7 min read
Jr Codex Generative AI Notes
Level: Intermediate Prerequisites: Chapter 2: Controlling the Output; NLP & LLM Notes, Module 7, Chapter 3 Time to complete: ~20 minutes
Table of Contents
- Why Structure Is the Whole Game
- Four Levels of Enforcement
- Schema-Constrained Output
- How Constrained Decoding Works
- Validate Beyond the Schema
- A Production-Shaped Extractor
- Summary & Next Steps
1. Why Structure Is the Whole Game
Module 1, Chapter 4 argued that reliable generative features are the ones with an automatic check on the output. Structured generation is that check, for every task where the consumer of the output is code rather than a person.
The Shift It Enables
─────────────────────────────────────────
UNSTRUCTURED model returns prose
you write a parser
the parser breaks whenever phrasing drifts
failures are SILENT and shaped like data
STRUCTURED model returns data matching a schema
validation is mechanical
failures are LOUD and catchable
─────────────────────────────────────────
2. Four Levels of Enforcement
These are genuinely different mechanisms with different guarantees, and the distinction matters.
Level 1 — ASKING
─────────────────────────────────────────
"Respond in JSON with keys name and age."
Guarantee: NONE. Usually works. Occasionally wrapped
in prose or a markdown fence. Always needs a parser
with a fallback.
Level 2 — JSON MODE
─────────────────────────────────────────
response_format={"type": "json_object"}
Guarantee: the output PARSES as JSON.
Not guaranteed: which keys exist, or their types.
Level 3 — SCHEMA-CONSTRAINED (structured outputs)
─────────────────────────────────────────
response_format={"type": "json_schema", ...}
Guarantee: the output MATCHES your schema — keys,
nesting, types, enums. Enforced during decoding, so
a violating token is never sampled.
Level 4 — TOOL / FUNCTION CALLING
─────────────────────────────────────────
Same enforcement as level 3, but framed as "call this
function with these arguments." Use when the model
must also DECIDE WHETHER to produce output at all, or
choose among several shapes.
→ The foundation of agents (Agentic AI Notes, Mod.2)
─────────────────────────────────────────
Default to level 3. Level 1 is for exploration only; level 2 is a legacy half-measure; level 4 is level 3 plus a decision.
3. Schema-Constrained Output
Define the schema once, in Python, and derive everything from it.
from pydantic import BaseModel, Field
from typing import Literal
class Invoice(BaseModel):
vendor: str = Field(description="Company that issued the invoice")
invoice_number: str
total_amount: float = Field(ge=0) # a business rule, in the schema
currency: Literal["USD", "EUR", "GBP", "INR"] # an ENUM, so the model cannot invent one
line_item_count: int = Field(ge=1)
response = client.chat.completions.parse(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "Extract invoice fields from the document."},
{"role": "user", "content": raw_document_text},
],
response_format=Invoice, # the SDK converts the model to a JSON schema
temperature=0, # extraction is a level-0-temperature task (Chapter 2)
)
invoice = response.choices[0].message.parsed # already a validated Invoice instance
print(invoice.total_amount, invoice.currency)Field Descriptions Are Prompt, Not Documentation
─────────────────────────────────────────
The `description` on every field is sent to the model
and is read as instruction. A vague field name plus an
empty description is a vague instruction.
Weak: date: str
Strong: issue_date: str = Field(
description="Invoice issue date as
YYYY-MM-DD. Use the issue date, not the
due date.")
─────────────────────────────────────────
4. How Constrained Decoding Works
Worth understanding, because it explains both the guarantee and its limits.
Masking, Step by Step
─────────────────────────────────────────
At each step the sampler knows which tokens the schema
ALLOWS given what has been emitted so far, and sets
every other token's logit to -infinity.
Emitted so far: {"currency":
Schema says: one of "USD","EUR","GBP","INR"
Allowed tokens: only " \"USD", " \"EUR", ...
The model CANNOT emit "XYZ". Not "is unlikely to" —
cannot. The token is masked out before sampling.
─────────────────────────────────────────
What This Does and Does Not Guarantee
─────────────────────────────────────────
GUARANTEED: valid JSON, correct keys, correct
types, values drawn from your enums
NOT GUARANTEED: that the values are CORRECT.
A constrained model faced with a missing field will
produce a well-formed, schema-valid, entirely
fabricated one — because the mask forces it to emit
SOMETHING of the right type.
─────────────────────────────────────────
That last point is the single most important thing in this chapter. Constrained decoding converts hallucination from a parsing problem into a data quality problem. It does not remove it.
The Mitigation: Make "Unknown" Representable
─────────────────────────────────────────
Give every uncertain field an escape hatch, so the
schema does not force fabrication:
vendor: str | None = None
confidence: Literal["high", "medium", "low"]
fields_not_found: list[str]
A model that CAN say "absent" often will. A model
that cannot, will invent.
─────────────────────────────────────────
5. Validate Beyond the Schema
The schema checks shape. You still have to check meaning.
from pydantic import field_validator, model_validator
class Invoice(BaseModel):
total_amount: float = Field(ge=0)
line_items: list[float]
issue_date: str
@field_validator("issue_date")
@classmethod
def iso_format(cls, v: str) -> str:
from datetime import date
date.fromisoformat(v) # raises on a malformed date
return v
@model_validator(mode="after")
def total_matches_items(self):
if abs(sum(self.line_items) - self.total_amount) > 0.01:
raise ValueError("line items do not sum to total") # CROSS-FIELD consistency
return selfThree Layers, Each Catching Something Different
─────────────────────────────────────────
1. SCHEMA is it the right shape?
2. FIELD RULES is each value individually sane?
3. CROSS-FIELD are the values consistent with
each other, and with the source?
Layer 3 catches the confident fabrications that
layers 1 and 2 wave through.
─────────────────────────────────────────
6. A Production-Shaped Extractor
from pydantic import ValidationError
def extract(client, document: str, schema: type[BaseModel], max_attempts: int = 3):
messages = [
{"role": "system", "content": "Extract the requested fields. "
"Use null for anything not present in the document."},
{"role": "user", "content": document},
]
for attempt in range(max_attempts):
response = client.chat.completions.parse(
model="gpt-4o-mini", messages=messages,
response_format=schema, temperature=0,
)
raw = response.choices[0].message
if raw.refusal: # the model declined — retrying will not help
raise ValueError(f"refused: {raw.refusal}")
try:
return raw.parsed # passed schema AND custom validators
except ValidationError as e:
# FEED THE ERROR BACK — the model can usually repair its own output
messages.append({"role": "assistant", "content": raw.content})
messages.append({"role": "user",
"content": f"That failed validation: {e}. Return corrected JSON."})
raise ValueError(f"failed validation after {max_attempts} attempts")Why Feeding the Error Back Works
─────────────────────────────────────────
The validation message is a precise, specific
description of what is wrong — far better instruction
than any prompt you could write in advance.
This "generate, validate, repair" loop is the
simplest useful form of self-correction, and it is
the direct ancestor of the reflection patterns in the
Agentic AI Notes, Module 4.
─────────────────────────────────────────
7. Summary & Next Steps
Key Takeaways
- Structured output is the automatic verification layer that makes code-consumed generation reliable; default to schema-constrained decoding, not prompt-and-parse.
- Constrained decoding masks disallowed tokens, so schema violations are impossible — but well-formed fabrication is not.
- Give uncertain fields a representable "unknown" (nullable, confidence, not-found lists), or the schema itself forces the model to invent.
- Validate in three layers — shape, per-field rules, cross-field consistency — and feed validation errors back for repair.
Concept Check
- What exactly does schema-constrained decoding guarantee, and what does it conspicuously not guarantee?
- Why can adding
vendor: str | None = Nonemeasurably reduce fabricated values? - Which validation layer would catch an invoice whose line items sum to 340 while the stated total is 430, and why do the other two miss it?
Next Chapter
→ Chapter 4: Context Windows & Long Inputs
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to Generative AI Index