Sequence Models
Why Sequences Need Special Architectures
Modules 2-4 covered fixed-size inputs: a tabular row of features, or a fixed-resolution image. Many important data types are fundamentally different — sequentia
Jr Codex Deep Learning Notes
Level: Intermediate–Advanced Prerequisites: Module 4: Convolutional Neural Networks Time to complete: ~15 minutes
Table of Contents
- What Makes Data "Sequential"
- Why CNNs and MLPs Fall Short
- The Variable-Length Problem
- What a Sequence Model Needs
- This Module's Roadmap
- Summary & Next Steps
1. What Makes Data "Sequential"
Modules 2-4 covered fixed-size inputs: a tabular row of features, or a fixed-resolution image. Many important data types are fundamentally different — sequential, where order matters and length varies: sentences (a variable number of words, in a specific order), time series (stock prices, sensor readings over time), audio waveforms, and DNA sequences.
Why Order Matters
─────────────────────────────────────────
"The dog bit the man" vs "The man bit the dog"
Identical WORDS, completely different MEANING — any model
that ignores word ORDER (like the ML Notes' bag-of-words
text representation) cannot distinguish these two sentences.
─────────────────────────────────────────
2. Why CNNs and MLPs Fall Short
The Core Mismatch
─────────────────────────────────────────
MLPs (Module 2): require a FIXED-SIZE input — a
sentence with 5 words and one with
50 words can't both be fed to the
same dense layer without awkward
padding/truncation that discards
useful structure
CNNs (Module 4): convolutional filters have a FIXED,
LOCAL receptive field — good for
images' local spatial patterns,
but sequences often have LONG-
RANGE dependencies (a pronoun
referring back to a noun many
words earlier) that a small local
filter can't directly capture
─────────────────────────────────────────
3. The Variable-Length Problem
A Concrete Illustration
─────────────────────────────────────────
"I love it." (3 words)
"I absolutely, completely, (11 words)
without question, love it."
Both are valid inputs to, say, a SENTIMENT classifier. A
fixed-size MLP input layer would need to accommodate the
LONGEST possible sentence, padding shorter ones — wasteful,
and it still doesn't solve the ORDER problem from Section 1.
─────────────────────────────────────────
4. What a Sequence Model Needs
Three Requirements This Module's Architectures Address
─────────────────────────────────────────
1. Handle VARIABLE-length input (Chapter 2's RNNs
naturally, without padding process one element
hacks: at a time, however
many there are)
2. Maintain some notion of ORDER (an RNN's hidden
and POSITION: state, updated
step by step,
naturally encodes
"what came before")
3. Capture dependencies across (Chapter 2-3
POTENTIALLY LONG distances: address this
directly; Module
6's attention
mechanism
addresses it far
more effectively)
─────────────────────────────────────────
5. This Module's Roadmap
─────────────────────────────────────────
Ch.1 (this chapter): the PROBLEM sequences pose
Ch.2: RNNs — the first architecture
designed specifically for
sequences
Ch.3: LSTMs/GRUs — fixing RNNs'
own limitations
Ch.4: Seq2Seq/Encoder-Decoder —
mapping one sequence TO
another (e.g. translation)
Ch.5: the remaining
limitations that
directly motivate
Module 6's attention
mechanism
─────────────────────────────────────────
A scope note: this module builds the general architectural foundation for sequence modeling. Applying these ideas specifically to natural language — tokenization, word embeddings, and language-specific models — is the focus of the separate NLP & LLM Notes; the sequences here are illustrated generically (numeric sequences, simplified text) to keep the focus on the architectural mechanics themselves.
6. Summary & Next Steps
Key Takeaways
- Sequential data (text, time series, audio) has variable length and order-dependent meaning, which fixed-size architectures like MLPs and CNNs handle poorly.
- MLPs require fixed-size inputs; CNNs' local receptive fields struggle with long-range dependencies common in sequences.
- A proper sequence model needs to handle variable length naturally, encode order/position, and capture both short- and long-range dependencies.
- This module builds general sequence modeling architecture; language-specific applications belong to the separate NLP & LLM Notes.
Concept Check
- Why can't a standard MLP directly handle sentences of varying length without padding or truncation?
- Give an example (other than the one in this chapter) where reordering a sequence changes its meaning entirely.
- What are the three requirements a proper sequence architecture needs to satisfy?
Next Chapter
→ Chapter 2: Recurrent Neural Networks
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to DL Index