NLP & LLMs

Decoder Models Gpt And Modern Llms

GPT: Architecture & Autoregressive Language Modeling

Module 4 covered BERT — an encoder-only model built for understanding a complete input at once. GPT (Generative Pre-trained Transformer), first released by Open

JrCodex·6 min read

Jr Codex NLP & LLM Notes

Level: Advanced Prerequisites: Module 4: Encoder Models — BERT & Friends Time to complete: ~25 minutes

Table of Contents

  1. A Different Family, a Different Goal
  2. GPT's Architecture
  3. Causal Language Modeling, Revisited
  4. Generating Text, Step by Step
  5. Decoding Strategies
  6. GPT in Practice
  7. Summary & Next Steps

1. A Different Family, a Different Goal

Module 4 covered BERT — an encoder-only model built for understanding a complete input at once. GPT (Generative Pre-trained Transformer), first released by OpenAI in 2018, is decoder-only, built specifically for generating text one token at a time.

2. GPT's Architecture

GPT is a stack of Transformer decoder blocks — but without the cross-attention sub-layer from DL Notes, Module 6, Chapter 4, Section 3, since there's no separate encoder to attend to. Each block contains only masked self-attention and a feed-forward sub-layer.

GPT's Block, Compared to BERT's (Module 4, Chapter 1)
─────────────────────────────────────────
  BERT encoder block:      UNMASKED self-attention (sees the
                              WHOLE input) + feed-forward
  GPT decoder block:          MASKED self-attention (sees ONLY
                                  prior tokens, DL Notes Module 6
                                  Ch.2 Sec.6) + feed-forward — NO
                                  cross-attention sub-layer (no
                                  separate encoder exists)
─────────────────────────────────────────

from transformers import GPT2Config, GPT2Model
 
config = GPT2Config(
    vocab_size=50257,          # GPT-2's BPE vocabulary (Module 3, Ch.1, Section 3)
    n_embd=768,                    # d_model
    n_layer=12,                        # stacked decoder blocks
    n_head=12,                             # multi-head attention (DL Notes, Module 6, Ch.2)
)
model = GPT2Model(config)
print(f"Total parameters: {sum(p.numel() for p in model.parameters()):,}")

3. Causal Language Modeling, Revisited

Module 3, Chapter 3, Section 4 introduced CLM generically. GPT is the model that CLM was specifically designed to train.

Why GPT's Masking Is Called "Causal"
─────────────────────────────────────────
  At position t, the model can only attend to positions
  1...t — never t+1 and beyond. This mirrors CAUSALITY in
  time: predicting the future (the next token) may only use
  information from the PAST and PRESENT, never information
  that hasn't "happened" yet in the sequence — directly the
  masking mechanism from DL Notes, Module 6, Chapter 2,
  Section 6.
─────────────────────────────────────────

import torch
 
def gpt_training_step(model, input_ids):
    # Labels are the SAME sequence, shifted by ONE position (Module 3, Ch.3, Section 4)
    labels = input_ids.clone()
 
    outputs = model(input_ids=input_ids, labels=labels)      # HuggingFace computes the shift AND
                                                                  # the cross-entropy loss internally
    return outputs.loss

4. Generating Text, Step by Step

This is where GPT genuinely differs from BERT in use, not just training: at inference time, GPT generates text autoregressively — directly the pattern from DL Notes, Module 5, Chapter 4, Section 5, now powered by a Transformer decoder instead of an RNN.

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
 
tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")
 
prompt = "The future of artificial intelligence is"
input_ids = tokenizer(prompt, return_tensors="pt")["input_ids"]
 
def generate_manually(model, input_ids, max_new_tokens=10):
    for _ in range(max_new_tokens):
        with torch.no_grad():
            outputs = model(input_ids)
        next_token_logits = outputs.logits[:, -1, :]      # predictions for the NEXT token ONLY
        next_token_id = torch.argmax(next_token_logits, dim=-1).unsqueeze(0)      # greedy choice (Section 5)
 
        input_ids = torch.cat([input_ids, next_token_id], dim=1)      # APPEND the new token, feed back IN
 
    return input_ids
 
generated = generate_manually(model, input_ids)
print(tokenizer.decode(generated[0]))
Why This Is Computationally Wasteful (Naively)
─────────────────────────────────────────
  Each new token requires re-processing the ENTIRE sequence so
  far — recomputing attention for EVERY previous token, EVERY
  time. Production inference systems use "KV caching" — storing
  each layer's KEY and VALUE vectors (DL Notes, Module 6, Ch.1,
  Section 3) from PREVIOUS steps, so only the NEWEST token needs
  fresh computation — covered practically in Module 9.
─────────────────────────────────────────

5. Decoding Strategies

The manual example above used greedy decoding — always picking the single most likely next token. Several alternative strategies exist, each trading off quality, diversity, and determinism differently.

from transformers import AutoTokenizer, AutoModelForCausalLM
 
tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")
input_ids = tokenizer("The weather today is", return_tensors="pt")["input_ids"]
 
# Greedy — always the single most likely token (deterministic, can be repetitive)
greedy_output = model.generate(input_ids, max_new_tokens=20, do_sample=False)
 
# Beam search — tracks several likely SEQUENCES simultaneously, picks the best overall
beam_output = model.generate(input_ids, max_new_tokens=20, num_beams=5, early_stopping=True)
 
# Temperature sampling — samples from the FULL probability distribution, scaled by "temperature"
temp_output = model.generate(input_ids, max_new_tokens=20, do_sample=True, temperature=0.7)
 
# Top-k sampling — samples only from the k MOST LIKELY tokens
topk_output = model.generate(input_ids, max_new_tokens=20, do_sample=True, top_k=50)
 
# Top-p (nucleus) sampling — samples from the SMALLEST set of tokens whose cumulative
# probability exceeds p — adapts the candidate pool size DYNAMICALLY per step
topp_output = model.generate(input_ids, max_new_tokens=20, do_sample=True, top_p=0.9)
Choosing a Decoding Strategy
─────────────────────────────────────────
  Greedy:      fast, deterministic, but often repetitive and
                 dull — rarely used for open-ended generation
  Beam search:     better for tasks with a "correct" answer
                       (translation, summarization) — LESS
                       suited to creative, open-ended text
  Temperature/         better for CREATIVE, diverse generation
  top-k/top-p             — LOWER temperature = more focused/
  sampling:                   deterministic; HIGHER temperature =
                                 more random/diverse; top-k/top-p
                                 prevent sampling extremely
                                 unlikely, nonsensical tokens
─────────────────────────────────────────

6. GPT in Practice

from transformers import pipeline
 
generator = pipeline("text-generation", model="gpt2")
 
result = generator(
    "In a world where AI writes its own code,",
    max_length=50,
    num_return_sequences=2,      # generate MULTIPLE candidate continuations
    temperature=0.8,
    top_p=0.9,
)
 
for i, output in enumerate(result):
    print(f"Generation {i+1}: {output['generated_text']}")

7. Summary & Next Steps

Key Takeaways

  • GPT is a stack of decoder blocks using only masked self-attention and feed-forward layers, without cross-attention, since there's no separate encoder.
  • Causal Language Modeling trains GPT to predict each next token using only prior context, matching exactly how it must generate text at inference time.
  • Text generation is autoregressive: each newly generated token is appended to the input and fed back in to generate the next one.
  • Decoding strategies (greedy, beam search, temperature/top-k/top-p sampling) trade off determinism, quality, and diversity differently depending on the task.

Concept Check

  1. Why does a GPT decoder block have no cross-attention sub-layer, unlike the original Transformer's decoder covered in the DL Notes?
  2. Why is naive autoregressive generation computationally wasteful, and what technique (mentioned briefly here) addresses it?
  3. When would you prefer beam search over temperature sampling, and vice versa?

Next Chapter

→ Chapter 2: Scaling Laws & the GPT Family's Evolution

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to NLP & LLM Index