Calling a Generative Model

Jr Codex Generative AI Notes

Level: Beginner Prerequisites: Module 1, Chapter 4; Python Notes, Module 3 Time to complete: ~20 minutes

Table of Contents

1. The Shape of Every Text API
2. Messages and Roles
3. A First Call
4. Streaming
5. Tokens and Cost
6. Failing Gracefully
7. Summary & Next Steps

1. The Shape of Every Text API

Providers differ in details, but the interface has converged. Learn it once and porting between them is a mechanical change.

The Common Interface
─────────────────────────────────────────
  IN:   a model name
        a list of MESSAGES (the conversation so far)
        sampling parameters (Chapter 2)

  OUT:  a message from the assistant
        a finish reason (why it stopped)
        a token usage count (what it cost)
─────────────────────────────────────────

The critical property, and the one that trips people up first:

The API Is STATELESS
─────────────────────────────────────────
  The server remembers NOTHING between calls.

  A "conversation" exists only because YOU resend the
  entire message history on every request. There is no
  session on the other end.

  Consequence: conversation length drives cost and
  latency linearly. Chapter 4 deals with this.
─────────────────────────────────────────

2. Messages and Roles

A request is a list of messages, each with a role.

A Conversation, on the Wire
─────────────────────────────────────────
  [
    {role: system,    "You are a terse SQL tutor."},
    {role: user,      "What does GROUP BY do?"},
    {role: assistant, "It collapses rows sharing a ..."},
    {role: user,      "Show me an example."}       ← new
  ]

  All four are sent. All four are billed as input.
─────────────────────────────────────────

The system message is the highest-leverage part of the request. It is the one place where instructions apply to every turn without being repeated, and models are trained to weight it heavily. Put format requirements, tone, and hard constraints there — not in the last user message where they compete with the actual question.

3. A First Call

import os
from openai import OpenAI
 
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])      # NEVER hardcode the key
 
response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "You explain Python concepts in two sentences."},
        {"role": "user", "content": "What is a list comprehension?"},
    ],
    max_tokens=200,            # a HARD CAP on output length — also a cost ceiling
)
 
print(response.choices[0].message.content)
print(response.choices[0].finish_reason)      # "stop" = it finished on its own
print(response.usage)                         # prompt_tokens, completion_tokens, total

Always Check finish_reason
─────────────────────────────────────────
  "stop"            the model finished naturally  ✓
  "length"          it hit max_tokens and was CUT OFF
                      mid-sentence — your output is
                      TRUNCATED, not complete
  "content_filter"  the provider blocked the output
  "tool_calls"      it wants to call a function

  Code that ignores finish_reason will silently ship
  half-finished text to users. This is the single most
  common bug in first LLM integrations.
─────────────────────────────────────────

4. Streaming

Generation is serial (Module 1, Ch.2), so a long response takes seconds. Streaming returns tokens as they are produced.

stream = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Explain the GIL in a paragraph."}],
    stream=True,
)
 
full = []
for chunk in stream:
    piece = chunk.choices[0].delta.content
    if piece:                            # the final chunk carries no content
        print(piece, end="", flush=True)
        full.append(piece)
 
text = "".join(full)                     # reassemble if you need to store it

Streaming Changes Nothing About Cost or Speed
─────────────────────────────────────────
  It does not make generation faster. It makes the WAIT
  visible instead of blank, which changes perceived
  latency dramatically.

  The trade: you cannot validate output before the user
  sees it. If you need to check a schema (Chapter 3) or
  filter content, DO NOT stream — or stream into a
  buffer and reveal only after validation.
─────────────────────────────────────────

5. Tokens and Cost

Billing is per token, split into input and output, and the two are priced differently — output typically costs several times more than input.

Estimating Before You Spend
─────────────────────────────────────────
  Rough rule for English:  1 token ≈ 4 characters
                           1 token ≈ 0.75 words

  Code, non-Latin scripts and unusual names tokenise
  far less efficiently. Measure rather than guess.
─────────────────────────────────────────

import tiktoken
 
enc = tiktoken.encoding_for_model("gpt-4o-mini")
n_tokens = len(enc.encode(long_document))
print(f"{n_tokens} input tokens")
 
# Cost scales with the WHOLE history, not just the newest message:
def conversation_tokens(messages, enc):
    return sum(len(enc.encode(m["content"])) for m in messages) + 4 * len(messages)

Three Cost Levers, in Order of Impact
─────────────────────────────────────────
  1. MODEL CHOICE      a small model is often 10-30x
                       cheaper. Most tasks in the
                       transform and extract patterns
                       (Module 1, Ch.4) do not need a
                       frontier model.

  2. HISTORY LENGTH    trim or summarise old turns
                       (Chapter 4). Cost grows with the
                       square of conversation length if
                       you never trim.

  3. max_tokens        capping output caps the
                       expensive half of the bill.
─────────────────────────────────────────

Module 7, Chapter 2 develops this into a full cost strategy including caching and model routing.

6. Failing Gracefully

Network calls to a shared, rate-limited service fail routinely. Treat failure as normal operation.

import time, random
from openai import RateLimitError, APIError
 
def call_with_retry(client, messages, model="gpt-4o-mini", max_attempts=4):
    for attempt in range(max_attempts):
        try:
            return client.chat.completions.create(model=model, messages=messages)
        except RateLimitError:
            if attempt == max_attempts - 1:
                raise
            wait = (2 ** attempt) + random.random()      # exponential backoff + JITTER
            time.sleep(wait)                             # jitter stops clients retrying in lockstep
        except APIError as e:
            if e.status_code and 400 <= e.status_code < 500:
                raise                                    # a BAD REQUEST will never succeed on retry
            time.sleep(2 ** attempt)
    raise RuntimeError("exhausted retries")

Retry What Is Transient, Not What Is Broken
─────────────────────────────────────────
  RETRY:      429 rate limit, 500/503, timeouts,
                connection resets
  DO NOT:     400 bad request, 401 bad key, 413 too
                large — these fail identically forever
                and burn your budget and latency
─────────────────────────────────────────

Set an explicit timeout on every client. The default in most SDKs is generous enough to hang a request thread for minutes.

7. Summary & Next Steps

Key Takeaways

• The API is stateless — conversation exists only because you resend the full history, so length drives cost and latency directly.
• The system message carries standing instructions and is weighted heavily; put format and constraints there rather than in the user turn.
• Always check finish_reason: a length finish means truncated output, and ignoring it silently ships broken text.
• Streaming improves perceived latency but removes your chance to validate before display; retry only transient errors, with backoff and jitter.

Concept Check

1. Why does a long conversation get more expensive per turn even when each new message is short?
2. What specifically goes wrong in an application that never inspects finish_reason?
3. You need to return validated JSON to a browser. What is the argument against streaming here, and what is the workaround?

Next Chapter

→ Chapter 2: Controlling the Output

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to Generative AI Index