Where Transformers Go From Here

Jr Codex Deep Learning Notes

Level: Advanced Prerequisites: Chapter 4: The Full Transformer Architecture Time to complete: ~15 minutes

What This Module Covered
Why Transformers Scale So Well
Beyond Language: Vision Transformers
The Compute Cost of Attention
Bridging to the NLP & LLM Notes
Summary & Next Steps

1. What This Module Covered

This Module's Arc
─────────────────────────────────────────
  Ch.1:      attention as a weighted, content-based lookup —
               query, key, value
  Ch.2:          self-attention (replacing recurrence entirely)
                   and multi-head attention (multiple
                   relationship types at once)
  Ch.3:              positional encoding — restoring the sense
                        of order that removing recurrence cost
  Ch.4:                   the full encoder/decoder Transformer,
                             and its encoder-only/decoder-only
                             variants
─────────────────────────────────────────

2. Why Transformers Scale So Well

Module 1, Chapter 3 mentioned that Transformers turned out to scale unusually well — larger models, more data, more compute, kept improving performance without the diminishing returns earlier architectures hit. This module's content explains the mechanism behind that observation:

Why Scale Keeps Helping
─────────────────────────────────────────
  Full parallelization (Ch.2):     unlike RNNs, a Transformer's
                                       training computation
                                       parallelizes almost
                                       completely across BOTH
                                       the batch dimension AND
                                       the sequence dimension —
                                       making it practical to
                                       train on FAR more data,
                                       in a FIXED amount of
                                       wall-clock time, than an
                                       equivalently-sized RNN
                                       ever could
  No long-range decay (Ch.1-2):        every position can attend
                                          to every other position
                                          directly — larger models
                                          don't lose the ability
                                          to use distant context
                                          the way deeper RNN
                                          stacks would
─────────────────────────────────────────

This combination — architecture that parallelizes well AND doesn't degrade with scale — is largely why Transformers, rather than ever-larger LSTMs, became the foundation for the scaling era described in Module 1.

3. Beyond Language: Vision Transformers

Module 4, Chapter 2 briefly previewed Vision Transformers (ViT) — applying this module's exact architecture to images, by treating an image as a sequence of fixed-size patches (analogous to how a sentence is a sequence of tokens), each projected into an embedding and processed identically to Chapter 4's encoder stack.

An Image as a "Sequence" (ViT's Core Idea)
─────────────────────────────────────────
  A 224x224 image, split into 16x16 patches, becomes a
  SEQUENCE of 196 patches (14 x 14 grid) — each patch is
  flattened and projected into a d_model-dimensional vector,
  exactly like a token embedding (Ch.3), with positional
  encoding added to preserve the patches' 2D spatial
  arrangement.

  From that point forward, it's an ORDINARY Transformer
  encoder (Ch.4, Section 2) — the SAME architecture used for
  text, applied to a sequence of image patches instead of
  word tokens.
─────────────────────────────────────────

This demonstrates something important: the Transformer architecture covered in this module isn't inherently language-specific — it's a general-purpose architecture for processing any data that can be represented as a sequence of tokens, which is precisely why it has spread far beyond its original translation use case.

4. The Compute Cost of Attention

A genuine limitation worth naming directly: self-attention's cost grows quadratically with sequence length — doubling the sequence length roughly quadruples the compute and memory required (since every position must compute a score against every other position).

Why This Matters in Practice
─────────────────────────────────────────
  A sequence of length N requires computing N × N attention
  scores (Ch.1, Section 4) — for SHORT sequences (a sentence),
  this is trivial; for VERY long sequences (an entire book, a
  long document), this quadratic cost becomes the dominant
  practical constraint on how much context a Transformer can
  process at once.

  This specific limitation has driven significant ongoing
  research (efficient/sparse attention variants, longer-
  context architectures) — a topic covered where it matters
  most practically, in the NLP & LLM Notes' discussion of
  context windows and long-context models.
─────────────────────────────────────────

5. Bridging to the NLP & LLM Notes

This curriculum has deliberately covered the Transformer architecture generically — the mechanics apply regardless of what kind of sequence is being processed. The separate NLP & LLM Notes picks up from exactly here, applying this architecture specifically to language:

What Carries Over DIRECTLY (No Need to Re-Learn)
─────────────────────────────────────────
  - Self-attention, multi-head attention (Ch.1-2)
  - Positional encoding (Ch.3)
  - Encoder-only vs decoder-only architectures (Ch.4, Sec.7)
  - Residual connections, layer normalization (Module 3),
      reused UNCHANGED inside every Transformer block
─────────────────────────────────────────

What's GENUINELY NEW in the NLP & LLM Notes
─────────────────────────────────────────
  - Tokenization: how raw text becomes the discrete tokens
      this module's architecture assumes as input
  - Pre-training objectives: masked language modeling (BERT),
      next-token prediction (GPT) — HOW these architectures
      are actually trained on raw text
  - Fine-tuning and prompt engineering: adapting a PRE-TRAINED
      model to a specific task, rather than training from
      scratch (directly extending Module 7's transfer learning
      concepts, covered next in THIS curriculum)
  - RAG, agents, and LLM-specific production concerns
─────────────────────────────────────────

6. Summary & Next Steps

Key Takeaways

Transformers scale unusually well because their training parallelizes almost completely, and their attention mechanism doesn't suffer the long-range decay that limited RNN-based architectures.
The Transformer architecture is general-purpose, not language-specific — Vision Transformers apply the identical architecture to sequences of image patches instead of word tokens.
Self-attention's compute cost grows quadratically with sequence length, which is the primary practical constraint on context length in modern models.
The NLP & LLM Notes builds directly on this module's architecture, adding tokenization, pre-training objectives, fine-tuning, and language-specific applications.

Concept Check

Why does the Transformer's parallelization advantage over RNNs matter so much for training on large datasets?
How does a Vision Transformer adapt the token-sequence architecture from this module to work with images?
Why does self-attention's compute cost grow quadratically with sequence length, and what practical limitation does this create?

Next Module

→ Module 7: Transfer Learning & Practical Training Strategies

Related Curriculum → NLP & LLM Notes (once Modules 7-9 of this curriculum are complete)

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to DL Index