Foundations Of Deep Learning
What Is Deep Learning & Why It Works
The AI Notes' Module 1 placed deep learning as a subfield of machine learning, which is itself a subfield of AI:
Jr Codex Deep Learning Notes
Level: Beginner Prerequisites: Machine Learning Notes, especially Module 1 (Foundations) and Module 3 (Regression) Time to complete: ~20 minutes
Table of Contents
- Where Deep Learning Sits in the Bigger Picture
- The Core Idea: Learned Feature Hierarchies
- Why "Deep" Matters
- What Changed to Make Deep Learning Practical
- What Deep Learning Is Good (and Bad) At
- How This Curriculum Is Organized
- Summary & Next Steps
1. Where Deep Learning Sits in the Bigger Picture
The AI Notes' Module 1 placed deep learning as a subfield of machine learning, which is itself a subfield of AI:
Artificial Intelligence
└── Machine Learning (ML Notes)
└── Deep Learning (THIS curriculum)
└── Specialized applications:
NLP/LLMs (separate curriculum)
Computer vision
Speech, reinforcement learning, etc.
Everything in the Machine Learning Notes — the bias-variance tradeoff, train/test splits, gradient descent, evaluation metrics — still applies here. Deep learning is not a different discipline with different rules; it's a specific family of models (neural networks) trained with the same underlying optimization ideas, scaled up.
2. The Core Idea: Learned Feature Hierarchies
Every algorithm in the ML Notes (linear regression, decision trees, SVMs) works on features you hand it. If you want a model to detect cats in photos using classical ML, you'd need to manually engineer features — edge detectors, color histograms, texture descriptors (ML Notes, Module 2) — before any classifier ever sees the data.
Deep learning's central innovation is letting the model learn the features themselves, directly from raw data, organized into layers of increasing abstraction:
A CNN Learning to Recognize a Face (Conceptual)
─────────────────────────────────────────
Layer 1 (early): learns to detect EDGES and simple
gradients — the most primitive
visual patterns
Layer 2: combines edges into TEXTURES and
simple SHAPES (curves, corners)
Layer 3: combines shapes into PARTS
(an eye, a nose, an ear)
Layer 4 (late): combines parts into
WHOLE OBJECTS (a face)
─────────────────────────────────────────
No human ever told the network "an eye looks like this." Each layer's features are discovered during training, purely by optimizing a loss function (ML Notes, Module 1, Chapter 3) on labeled examples.
3. Why "Deep" Matters
"Deep" refers to having many layers stacked on top of each other. This depth is what makes hierarchical feature learning possible.
# A SHALLOW network — one hidden layer
# Can only combine inputs in relatively simple ways
input -> [hidden layer] -> output
# A DEEP network — many hidden layers
# Each layer builds on the abstractions of the layer before it
input -> [layer 1] -> [layer 2] -> [layer 3] -> ... -> [layer N] -> outputA mathematical result called the universal approximation theorem says that even a single, sufficiently wide hidden layer can, in principle, approximate any continuous function. In practice, going deep (many moderately-sized layers) rather than wide (one enormous layer) tends to learn better, more efficient, more generalizable representations for complex, structured data like images, audio, and text — which is why "deep learning" is the term that stuck, not "wide learning."
4. What Changed to Make Deep Learning Practical
Neural networks themselves date back to the 1950s-80s (covered in Chapter 3), but they were largely dismissed as impractical until roughly 2012. Three things changed:
The Three Enablers
─────────────────────────────────────────
1. DATA: The internet produced enormous labeled
datasets (ImageNet: 14 million labeled
images) — deep networks are extremely
data-hungry, and this data simply didn't
exist at scale before
2. COMPUTE: GPUs, originally built for rendering video
game graphics, turned out to be extremely
good at the parallel matrix multiplications
deep networks require — training that took
weeks on CPUs takes hours on GPUs
3. ALGORITHMS: Better weight initialization, activation
functions (ReLU replacing sigmoid),
and regularization techniques
(covered in Module 3) solved training
instabilities that had stalled progress
for decades
─────────────────────────────────────────
5. What Deep Learning Is Good (and Bad) At
Deep Learning Tends to WIN When:
─────────────────────────────────────────
- Data is unstructured: images, audio, raw text, video
- There's a LARGE amount of training data available
- The underlying patterns are complex and hierarchical
- Manual feature engineering is difficult or infeasible
─────────────────────────────────────────
Classical ML (ML Notes) Often Still Wins When:
─────────────────────────────────────────
- Data is tabular/structured (spreadsheet-like columns)
— gradient-boosted trees (ML Notes, Module 5) frequently
outperform deep networks here
- The dataset is SMALL — deep networks tend to overfit
badly without enough examples (revisit the bias-variance
discussion in ML Notes, Module 1, Chapter 4)
- Interpretability is critical — a decision tree's logic
is inspectable; a 50-layer network's is not
- Training/inference compute budget is tightly constrained
─────────────────────────────────────────
This is a genuinely important, practical distinction: reaching for deep learning by default, even for a small tabular dataset, is a common beginner mistake. The ML Notes, Module 1, Chapter 3 workflow — start simple, only add complexity that's earned — still applies.
6. How This Curriculum Is Organized
The Nine Modules Ahead
─────────────────────────────────────────
1. Foundations — this module: what, why, history
2. Neural Networks from Scratch — the math and mechanics
3. Training Deep Networks — making training actually work
4. Convolutional Neural Networks — vision
5. Sequence Models — RNNs/LSTMs for
ordered data
6. Attention & Transformers — the architecture
behind modern AI
7. Transfer Learning — reusing what
others trained
8. Generative Deep Learning — models that
CREATE data
9. Engineering & Capstone — building
real systems
─────────────────────────────────────────
A scope note: this curriculum covers deep learning architectures and training in depth — the same neural network foundations apply whether the input is an image, audio, or text. Applying these foundations specifically to language (word embeddings, BERT, GPT, prompt engineering, fine-tuning LLMs) is the focus of the separate NLP & LLM Notes, which builds directly on Module 6 of this curriculum.
7. Summary & Next Steps
Key Takeaways
- Deep learning is a subfield of machine learning built on neural networks; every ML foundation (bias-variance, train/test splits, gradient descent) still applies.
- Its core innovation is learning hierarchical features directly from raw data, rather than requiring hand-engineered features.
- "Deep" (many stacked layers) enables learning increasingly abstract representations, layer by layer.
- Deep learning's practical rise (~2012 onward) was driven by three enablers: massive labeled datasets, GPU compute, and algorithmic fixes to training instability.
- Deep learning isn't automatically the right tool — classical ML often wins on small or tabular datasets, or when interpretability matters.
Concept Check
- Why can't a classical ML algorithm like linear regression "learn features" the way a deep neural network can?
- What does depth (many layers) provide that a single very wide layer does not, in practice?
- Name the three enablers that made deep learning practical starting around 2012.
Next Chapter
→ Chapter 2: From Biological to Artificial Neurons
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to DL Index