NLP & LLMs

Prompt Engineering

Structured Output & Function Calling

Every example in Chapters 1-2 produced free-form natural language text. Real applications often need an LLM's output to be parsed programmatically by other soft

JrCodex·7 min read

Jr Codex NLP & LLM Notes

Level: Advanced Prerequisites: Chapter 2: Advanced Prompting — Chain-of-Thought & Few-Shot Techniques Time to complete: ~25 minutes

Table of Contents

  1. Why Free-Form Text Isn't Always Enough
  2. Prompting for JSON Output
  3. Constrained Generation
  4. Function Calling — the Core Idea
  5. A Complete Function-Calling Example
  6. Why Function Calling Matters — the Bridge to Agents
  7. Summary & Next Steps

1. Why Free-Form Text Isn't Always Enough

Every example in Chapters 1-2 produced free-form natural language text. Real applications often need an LLM's output to be parsed programmatically by other software — which requires the output to follow a strict, predictable structure, not just be well-written prose.

2. Prompting for JSON Output

The simplest approach: explicitly instruct the model to respond in JSON, ideally with an example.

structured_prompt = """Extract the following information from the review below,
and respond ONLY with valid JSON in this exact format:
{"product": "...", "sentiment": "positive/negative/neutral", "rating_mentioned": true/false}
 
Review: "I bought this blender last month and it's been fantastic for smoothies,
though a bit loud. Would recommend!"
"""
 
import json
 
def parse_llm_json_response(response_text):
    try:
        return json.loads(response_text)
    except json.JSONDecodeError:
        print("WARNING: model output was not valid JSON — needs handling (Section 3)")
        return None
Why This ALONE Isn't Fully Reliable
─────────────────────────────────────────
  An LLM generates text TOKEN BY TOKEN (Module 5, Ch.1) with
  NO built-in guarantee of syntactic correctness — it might
  add explanatory text before/after the JSON, use SINGLE
  quotes instead of double quotes, or otherwise produce text
  that FAILS to parse as valid JSON, despite being instructed
  clearly. Section 3 covers a more RELIABLE alternative.
─────────────────────────────────────────

3. Constrained Generation

Many modern LLM APIs and libraries support constrained decoding (or "structured outputs") — directly restricting which tokens can be generated at each step, GUARANTEEING valid output structure, rather than just hoping the model follows instructions.

from pydantic import BaseModel
from openai import OpenAI
 
class ProductReview(BaseModel):
    product: str
    sentiment: str
    rating_mentioned: bool
 
client = OpenAI()
response = client.beta.chat.completions.parse(
    model="gpt-4",
    messages=[{"role": "user", "content": "Extract info from: 'Great blender, a bit loud though.'"}],
    response_format=ProductReview,      # the API guarantees output MATCHING this schema
)
parsed = response.choices[0].message.parsed
print(parsed.product, parsed.sentiment, parsed.rating_mentioned)
How Constrained Decoding Actually Works
─────────────────────────────────────────
  At EACH generation step (Module 5, Chapter 1, Section 4's
  autoregressive loop), the decoding process CHECKS which
  next tokens would keep the output VALID according to the
  target schema (e.g., after {"product": ", only a STRING
  value or specific tokens are allowed next) — and MASKS OUT
  (sets the probability to zero for) any token that would
  violate the schema, directly analogous to Module 5, Chapter
  1's masking mechanism, but applied for STRUCTURAL validity
  rather than causality.
─────────────────────────────────────────

This provides a genuine guarantee that free-form prompting (Section 2) cannot — the output is structurally valid by construction, not merely "likely to be valid if the model follows instructions well."

4. Function Calling — the Core Idea

Function calling (also called "tool calling") extends structured output one step further: instead of just extracting data, the model decides which function to call and with what arguments, based on the user's natural language request — the model itself never executes anything; it only decides intent and parameters.

function_definitions = [
    {
        "name": "get_weather",
        "description": "Get the current weather for a specific location",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {"type": "string", "description": "City name"},
                "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
            },
            "required": ["location"],
        },
    }
]
 
response = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "What's the weather like in Tokyo right now?"}],
    tools=[{"type": "function", "function": f} for f in function_definitions],
)
 
# The model responds with a STRUCTURED function call, NOT free text:
# {"name": "get_weather", "arguments": {"location": "Tokyo", "unit": "celsius"}}
The Critical Separation of Concerns
─────────────────────────────────────────
  The LLM's ONLY job: understand the request and produce a
  VALID function call (WHAT to do, with WHAT parameters) —
  directly Section 3's constrained generation, applied to
  function signatures instead of arbitrary schemas.

  The APPLICATION's job: actually EXECUTE the function (e.g.
  call a real weather API), then feed the RESULT back to the
  model as additional context, for it to compose a final,
  natural-language response.
─────────────────────────────────────────

5. A Complete Function-Calling Example

import json
 
def get_weather(location, unit="celsius"):
    """A REAL implementation would call an actual weather API"""
    return {"location": location, "temperature": 18, "unit": unit, "condition": "cloudy"}
 
def handle_conversation_turn(client, user_message):
    messages = [{"role": "user", "content": user_message}]
 
    response = client.chat.completions.create(
        model="gpt-4", messages=messages,
        tools=[{"type": "function", "function": f} for f in function_definitions],
    )
 
    message = response.choices[0].message
 
    if message.tool_calls:      # the model DECIDED a function call was appropriate
        tool_call = message.tool_calls[0]
        function_args = json.loads(tool_call.function.arguments)
 
        # The APPLICATION executes the ACTUAL function (Section 4's separation of concerns)
        result = get_weather(**function_args)
 
        messages.append(message)
        messages.append({
            "role": "tool", "tool_call_id": tool_call.id, "content": json.dumps(result),
        })
 
        # Feed the RESULT back, letting the model compose a natural final response
        final_response = client.chat.completions.create(model="gpt-4", messages=messages)
        return final_response.choices[0].message.content
 
    return message.content      # no function call needed — respond directly
 
print(handle_conversation_turn(client, "What's the weather in Tokyo?"))

6. Why Function Calling Matters — the Bridge to Agents

Function calling is the exact mechanism underlying Module 8's agent systems — an "agent," in the modern LLM sense, is fundamentally a loop: the model decides which tool to call, the application executes it, the result is fed back, and the model decides the next action (possibly another tool call), repeating until the task is complete.

The Direct Connection Forward
─────────────────────────────────────────
  THIS chapter:      a SINGLE function call, decided and
                        executed once
  Module 8's agents:      a LOOP of potentially MANY function
                             calls, where the model decides,
                             after EACH result, whether the task
                             is complete or another tool call is
                             needed — directly building on this
                             chapter's mechanism, repeated and
                             chained
─────────────────────────────────────────

This also directly connects to the AI Notes' agent framework — perceive (the user's request and tool results), decide (which action/tool to take), act (execute it) — the SAME agent loop, now implemented with an LLM as the decision-making component instead of the classical, rule-based agents covered there.

7. Summary & Next Steps

Key Takeaways

  • Free-form text prompting for structured data (like JSON) is not fully reliable, since LLMs have no inherent guarantee of syntactic correctness token by token.
  • Constrained decoding masks out invalid next tokens at each generation step, guaranteeing structurally valid output by construction, rather than merely instructing the model to comply.
  • Function calling extends structured output to let a model decide which function to call and with what arguments, while the application (not the model) actually executes it.
  • Function calling is the exact mechanism underlying agent systems — a single function call becomes a repeated loop when chained across multiple decision points.

Module 7 Complete — What's Next

You've now covered the complete prompting toolkit: fundamentals, advanced reasoning techniques, and structured/function-calling output — all without changing a single model weight. Module 8 covers grounding LLMs in external, up-to-date knowledge (RAG) and extends function calling into full agent systems that can take multi-step actions.

Concept Check

  1. Why is instructing a model to "respond in JSON" not a fully reliable guarantee of valid JSON output?
  2. In function calling, what is the model responsible for deciding, and what is the application responsible for actually doing?
  3. How does a single function call (this chapter) relate to a full agent loop (Module 8)?

Next Module

→ Module 8: RAG & Agents

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to NLP & LLM Index