Decoder Models Gpt And Modern Llms
In-Context Learning & Emergent Abilities
Every model in the ML Notes and DL Notes up to this point learns by adjusting weights via gradient descent (DL Notes, Module 2, Chapter 5). In-context learning
Jr Codex NLP & LLM Notes
Level: Advanced Prerequisites: Chapter 2: Scaling Laws & the GPT Family's Evolution Time to complete: ~20 minutes
Table of Contents
- A Genuinely Novel Capability
- Zero-Shot, One-Shot, and Few-Shot
- Why In-Context Learning Works
- In-Context Learning in Practice
- Emergent Abilities
- The Debate Around "Emergence"
- Summary & Next Steps
1. A Genuinely Novel Capability
Every model in the ML Notes and DL Notes up to this point learns by adjusting weights via gradient descent (DL Notes, Module 2, Chapter 5). In-context learning is different: a large enough pretrained model can perform a new task without any weight updates at all, purely by being shown examples within its input prompt.
2. Zero-Shot, One-Shot, and Few-Shot
Three Levels of In-Context Learning
─────────────────────────────────────────
Zero-shot: the task is described in NATURAL LANGUAGE,
with NO examples provided at all
One-shot: the task is described, with EXACTLY ONE
example demonstrating it
Few-shot: the task is described, with SEVERAL
(typically 2-10+) examples
demonstrating it
─────────────────────────────────────────
# Zero-shot
zero_shot_prompt = """Classify the sentiment of this review as positive or negative.
Review: "This product exceeded all my expectations!"
Sentiment:"""
# Few-shot
few_shot_prompt = """Classify the sentiment of each review as positive or negative.
Review: "Terrible quality, broke after one use."
Sentiment: negative
Review: "Absolutely love it, best purchase this year!"
Sentiment: positive
Review: "Works fine but nothing special."
Sentiment: neutral
Review: "This product exceeded all my expectations!"
Sentiment:"""
from transformers import pipeline
generator = pipeline("text-generation", model="gpt2-large")
result = generator(few_shot_prompt, max_new_tokens=5)
print(result[0]["generated_text"])Why Few-Shot Typically Outperforms Zero-Shot
─────────────────────────────────────────
The examples in the prompt demonstrate the EXACT desired
output FORMAT and the SPECIFIC task boundaries (e.g. is
"neutral" a valid label?) more precisely than a natural
language DESCRIPTION alone can convey — directly analogous
to how a few worked examples often clarify instructions
better than an abstract explanation, for humans too.
─────────────────────────────────────────
3. Why In-Context Learning Works
There is no separate, dedicated "in-context learning module" anywhere in GPT's architecture (Chapter 1) — this capability is a direct consequence of self-attention and massive-scale CLM pretraining.
The Leading Explanation
─────────────────────────────────────────
During pretraining on TRILLIONS of tokens (Chapter 2,
Section 6), the model encounters COUNTLESS naturally-
occurring examples of PATTERN-following text: FAQs, worked
examples in textbooks, structured lists, tables, and other
text where earlier content PREDICTS the structure/content of
later content.
Self-attention (DL Notes, Module 6, Ch.2) lets the model
directly ATTEND to the examples provided in the CURRENT
prompt, and — having learned FROM its training data that
earlier examples in a sequence often predict how LATER
entries should look — apply the SAME "continue the pattern"
skill it learned generally, to the SPECIFIC pattern shown in
this particular prompt.
─────────────────────────────────────────
An Important Distinction
─────────────────────────────────────────
In-context learning does NOT update the model's WEIGHTS
(unlike DL Notes, Module 7's fine-tuning) — the model's
parameters are IDENTICAL before and after processing a
few-shot prompt. What changes is only the ACTIVATIONS
computed for THIS SPECIFIC input — the "learning" is
entirely contained within a single forward pass's
attention computations, and vanishes the moment the
conversation/context ends.
─────────────────────────────────────────
4. In-Context Learning in Practice
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("gpt2-large")
model = AutoModelForCausalLM.from_pretrained("gpt2-large")
def few_shot_classify(examples, query, labels):
prompt = ""
for text, label in examples:
prompt += f"Text: {text}\nLabel: {label}\n\n"
prompt += f"Text: {query}\nLabel:"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=3, do_sample=False)
generated = tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:])
return generated.strip()
examples = [
("The movie was a total waste of time.", "negative"),
("I really enjoyed every minute of it.", "positive"),
]
prediction = few_shot_classify(examples, "Best film I've seen all year!", labels=["positive", "negative"])
print(f"Prediction: {prediction}")5. Emergent Abilities
An emergent ability is a capability that appears to arise suddenly at a certain model scale, essentially absent in smaller models, rather than improving smoothly and gradually (as Chapter 2's scaling laws predict for loss itself).
Documented Examples of Claimed Emergence
─────────────────────────────────────────
Multi-step arithmetic: small models perform near
RANDOM chance; beyond a certain
scale threshold, accuracy jumps
substantially
Chain-of-thought reasoning (covered in full in Module 7)
benefits: — smaller models see LITTLE
benefit from being
prompted to "think step
by step"; larger models
benefit dramatically
Instruction following understanding and
without explicit correctly executing
fine-tuning: NOVEL, complex
natural-language
instructions
─────────────────────────────────────────
6. The Debate Around "Emergence"
Worth knowing as a genuinely open, actively debated research question: some researchers argue that apparent "emergence" is partly an artifact of measurement — using metrics that are inherently discontinuous (e.g. exact-match accuracy on multi-step arithmetic) makes GRADUAL underlying improvement look like a sudden jump, while smoother metrics applied to the SAME underlying capability may show continuous, predictable improvement all along.
Why This Debate Matters Practically
─────────────────────────────────────────
If abilities TRULY emerge unpredictably at scale, planning
what a NEXT-GENERATION, larger model might be capable of
becomes GENUINELY difficult to forecast in advance. If
apparent emergence is largely a MEASUREMENT artifact, then
Chapter 2's scaling laws may extend further than they first
appear to — a smoother, more PREDICTABLE picture than the
"sudden capability jump" framing suggests.
─────────────────────────────────────────
This remains an active area of research — treat "emergent abilities" as an evocative, useful description of an observed pattern, not settled scientific consensus about an underlying mechanism.
7. Summary & Next Steps
Key Takeaways
- In-context learning lets a sufficiently large pretrained model perform new tasks from examples shown in its prompt, without any weight updates — a fundamentally different mechanism than fine-tuning.
- Few-shot prompting typically outperforms zero-shot by demonstrating the exact desired output format and task boundaries directly, rather than relying on natural language description alone.
- In-context learning is believed to emerge from self-attention combined with massive-scale exposure to naturally pattern-following text during pretraining.
- Emergent abilities — capabilities appearing suddenly at certain scales — are a documented but actively debated phenomenon, with some researchers attributing apparent emergence partly to discontinuous evaluation metrics.
Concept Check
- What is the key difference between in-context learning and the fine-tuning covered in the DL Notes?
- Why does few-shot prompting typically outperform zero-shot prompting?
- What is the core argument in the debate about whether "emergent abilities" are genuine or a measurement artifact?
Next Chapter
→ Chapter 4: The Modern LLM Landscape
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to NLP & LLM Index