A Brief History of Deep Learning

Jr Codex Deep Learning Notes

Level: Beginner Prerequisites: Chapter 2: From Biological to Artificial Neurons Time to complete: ~15 minutes

Why History Matters Here
1958 — The Perceptron
1969-1986 — The First AI Winter
1986 — Backpropagation Popularized
1990s-2000s — The Second Lull
2012 — The ImageNet Moment
2017 — Attention Is All You Need
2018-Present — The Scaling Era
Summary & Next Steps

1. Why History Matters Here

Deep learning's history isn't trivia — it directly explains why the field is structured the way it is: why certain techniques (Chapter 3's ReLU, Module 3's batch normalization) exist, and why enthusiasm for neural networks has cycled between hype and skepticism more than once. Understanding this cycle builds realistic expectations rather than assuming the current wave of progress is either the first or the last.

2. 1958 — The Perceptron

Frank Rosenblatt built the Perceptron — a single artificial neuron (Chapter 2), with a learning rule that could adjust its weights based on mistakes. It generated enormous excitement; the New York Times reported the Navy believed it would eventually "walk, talk, see, write, reproduce itself and be conscious of its existence."

3. 1969-1986 — The First AI Winter

In 1969, Marvin Minsky and Seymour Papert published Perceptrons, mathematically proving that a single-layer perceptron cannot solve non-linearly separable problems like XOR (Chapter 2, Section 4). Funding and interest collapsed for over a decade — this period is often called the first AI winter.

The Core Problem
─────────────────────────────────────────
  Multi-layer networks (which CAN solve XOR) were understood
  in principle, but nobody had an efficient algorithm to
  TRAIN their weights — with no way to adjust hidden layer
  weights based on the final error, multi-layer perceptrons
  remained a theoretical curiosity, not a practical tool.
─────────────────────────────────────────

4. 1986 — Backpropagation Popularized

Rumelhart, Hinton, and Williams published a paper popularizing backpropagation — an efficient algorithm for computing how much each weight in a multi-layer network contributed to the final error, using the chain rule of calculus (covered in full in Module 2, Chapter 4). This solved the exact problem that stalled the field in 1969, and briefly reignited interest.

5. 1990s-2000s — The Second Lull

Despite backpropagation, deep networks remained hard to train in practice — gradients would often "vanish" or "explode" across many layers (a problem covered in Module 3, Chapter 6), and the field lacked the data and compute (Chapter 1, Section 4) to demonstrate clear wins. Support Vector Machines and other classical ML methods, requiring far less data and compute, dominated during this period instead.

6. 2012 — The ImageNet Moment

AlexNet, a deep convolutional neural network (Module 4), won the 2012 ImageNet image classification competition by a stunning margin — roughly 10 percentage points better than the next-best (non-deep-learning) approach. This is widely regarded as the moment deep learning went from academic curiosity to dominant paradigm.

Why AlexNet Won, Connecting Back to Chapter 1's Three Enablers
─────────────────────────────────────────
  Data:      trained on ImageNet's 1.2 million labeled images
  Compute:      trained on GPUs, making the training TIME
                   practical (days instead of months)
  Algorithms:       used ReLU activations (Module 2) instead of
                       sigmoid, and dropout (Module 3) — both
                       addressing training problems that had
                       stalled the field for decades
─────────────────────────────────────────

The years following 2012 saw rapid architectural progress: VGGNet, GoogLeNet, and ResNet (all covered in Module 4) each pushed image classification accuracy further.

7. 2017 — Attention Is All You Need

Google researchers published "Attention Is All You Need," introducing the Transformer architecture (Module 6) — initially for machine translation. It replaced the sequential processing of RNNs (Module 5) with a mechanism that could process entire sequences in parallel while still capturing long-range relationships, dramatically improving both training speed and quality on sequence tasks.

8. 2018-Present — The Scaling Era

Transformers turned out to scale remarkably well — larger models, trained on more data, with more compute, kept improving performance, without hitting the diminishing returns earlier architectures showed. This observation (later formalized as "scaling laws") drove the development of increasingly large Large Language Models (BERT in 2018, GPT-2/GPT-3 in 2019-2020, and their successors) — the subject of the separate NLP & LLM Notes, which builds directly on this curriculum's Module 6.

The Pattern Across All Three Breakthroughs (2012, 2017, Scaling Era)
─────────────────────────────────────────
  Each breakthrough followed the SAME formula from Chapter 1:
  more data + more compute + an algorithmic/architectural
  insight that made using both practical. This pattern is
  worth watching for as the field continues to evolve.
─────────────────────────────────────────

9. Summary & Next Steps

Key Takeaways

Neural networks have cycled through hype and "AI winters" more than once — the field's current momentum is its third major wave, not its first.
The first winter (1969-1986) was caused by an inability to train multi-layer networks; backpropagation (1986) solved this in principle, but data/compute limitations caused a second lull.
The 2012 ImageNet moment (AlexNet) marked deep learning's decisive practical breakthrough, driven by the same three enablers covered in Chapter 1: data, compute, and algorithms.
The 2017 Transformer architecture and the subsequent scaling era directly led to the modern large language models covered in the separate NLP & LLM Notes.

Concept Check

What specific mathematical limitation caused the first AI winter, and what 1986 development addressed it?
What three factors is the 2012 AlexNet breakthrough usually attributed to?
What architectural shift, introduced in 2017, underlies most modern large language models?

Next Chapter

→ Chapter 4: The Deep Learning Landscape & Tooling

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to DL Index