Reinforcement Learning For ML Practitioners
From Tabular RL to Deep RL
The AI Notes' Module 5, Chapter 5 ended by identifying tabular Q-learning's key limitation: a q_table dictionary works for small, discrete state spaces, but bec
Jr Codex ML Notes
Level: Advanced Prerequisites: Chapter 3: Actor-Critic Methods Time to complete: ~20 minutes
Table of Contents
- The Scaling Problem, Revisited
- Function Approximation — the General Fix
- Deep Q-Networks (DQN)
- Deep Policy Gradients and Deep Actor-Critic
- Why This Requires the Deep Learning Notes
- A Map of Where Everything Fits
- Summary & Next Steps
1. The Scaling Problem, Revisited
The AI Notes' Module 5, Chapter 5 ended by identifying tabular Q-learning's key limitation: a q_table dictionary works for small, discrete state spaces, but becomes impossibly large for complex environments (millions of possible pixel configurations, continuous sensor readings). This exact limitation applies equally to Chapters 2-3's actor and critic — both were represented here as simple linear functions (state @ weights), which have their own real limits.
import numpy as np
# A LINEAR policy/critic (Chapters 2-3) can only represent LINEAR
# relationships between state features and action probabilities/values —
# exactly Module 4, Chapter 1's logistic regression limitation, revisited
# For a SIMPLE environment (a few hand-crafted features), linear works fine.
# For RAW PIXELS from a video game screen (thousands of numbers, complex
# non-linear relationships), a linear function is FAR too limited —
# exactly like Module 4, Chapter 1's decision boundary limitation motivated
# Chapters 2-4's more flexible classification algorithms.2. Function Approximation — the General Fix
Rather than a lookup table (AI Notes) or a simple linear function (Chapters 2-3), deep reinforcement learning uses a neural network (full depth in the Deep Learning Notes) to approximate the Q-function, policy, or value function.
The Progression
─────────────────────────────────────────
Tabular (AI Notes): Q(s,a) stored EXPLICITLY, one
entry per state-action pair —
works only for SMALL, discrete
state spaces
Linear function Q(s,a), π(a|s), or V(s)
approximation computed as a LINEAR
(Chapters 2-3): combination of state
features — works for
MODERATE complexity
Deep function Q(s,a), π(a|s), or V(s)
approximation computed by a NEURAL
(this chapter, NETWORK — can represent
Deep Learning Notes): ARBITRARILY complex,
non-linear relationships,
directly from raw
input (pixels, sensor
arrays)
─────────────────────────────────────────
3. Deep Q-Networks (DQN)
The AI Notes previewed this directly: DeepMind's famous Atari-playing system replaced the Q-table with a neural network taking raw pixels as input and outputting Q-values for each possible action.
DQN, Conceptually
─────────────────────────────────────────
Input: raw pixel array (a game screen)
│
▼
[Neural Network] (convolutional layers — full depth
│ in the Deep Learning Notes' CNN chapter)
▼
Output: one Q-value PER POSSIBLE ACTION
(e.g. "move left": 12.4, "move right": 8.1,
"jump": 15.7, ...)
The Q-LEARNING UPDATE RULE (AI Notes, Module 5, Ch.5) stays
EXACTLY the same — only the REPRESENTATION of Q(s,a) changed,
from a table lookup to a neural network's forward pass.
─────────────────────────────────────────
Key Innovations DQN Needed to Make This Work
─────────────────────────────────────────
Experience Replay: store past (state, action, reward,
next_state) transitions in a buffer,
and train on RANDOM SAMPLES from it
— breaks harmful correlations
between consecutive experiences
(directly echoes why Module 1,
Chapter 5's shuffled train/test
splits matter for i.i.d. assumptions)
Target Networks: use a SEPARATE, slowly-updated
copy of the network to compute
the Q-LEARNING TARGET — stabilizes
training, since the target would
otherwise shift every single
step, alongside the network being
trained (a moving-target problem)
─────────────────────────────────────────
4. Deep Policy Gradients and Deep Actor-Critic
The exact same substitution applies to Chapters 2-3's methods — replace the linear actor/critic with a neural network.
# Conceptual sketch — the ACTOR-CRITIC update logic from Chapter 3
# stays STRUCTURALLY IDENTICAL; only the actor/critic's INTERNAL
# representation changes
# Chapter 3's linear actor:
# scores = state @ weights
# Deep RL's neural network actor (full mechanics in the Deep Learning Notes):
# scores = neural_network.forward(state) # a multi-layer network,
# not a single matrix multiplyWhat Stays the Same vs What Changes
─────────────────────────────────────────
STAYS THE SAME: the RL algorithm's LOGIC — Q-learning's
update rule (AI Notes), REINFORCE's
gradient direction (Ch.2), actor-critic's
TD error mechanism (Ch.3)
CHANGES: HOW Q(s,a), π(a|s), or V(s) are
COMPUTED — from a table, to a
linear function, to a deep neural
network — each step handling
MORE complex input/relationships
─────────────────────────────────────────
5. Why This Requires the Deep Learning Notes
This chapter deliberately stops at the conceptual bridge — building and training an actual neural network (forward propagation, backpropagation, convolutional layers for image input, training stability techniques) is the entire subject of the dedicated Deep Learning Notes, which this module doesn't re-derive.
The Handoff, Explicitly
─────────────────────────────────────────
THIS module gave you: the RL algorithms themselves
(value-based from the AI Notes,
policy-based/actor-critic from
Chapters 1-3) — the LOGIC of
how an agent learns from reward
The Deep Learning Notes gives you: how to actually BUILD
will give you: and TRAIN the neural networks
that make Deep RL practical
at scale (neural network
fundamentals, backpropagation,
CNNs for image-based states,
training stability techniques)
Combining BOTH is what production Deep RL systems (DQN,
AlphaGo, modern RLHF-trained LLMs) actually consist of.
─────────────────────────────────────────
6. A Map of Where Everything Fits
The Complete Reinforcement Learning Picture, Across This Curriculum
─────────────────────────────────────────
AI Notes, Module 5: agent architectures, MDPs,
Q-learning (value-based,
TABULAR)
ML Notes, Module 8 (HERE): policy gradients, actor-critic
(POLICY-BASED, still mostly
LINEAR function approximation)
Deep Learning Notes: neural network fundamentals
— the missing piece for
SCALING both value-based
(DQN) and policy-based/
actor-critic methods to
real-world complexity
NLP/LLM Notes: RLHF — actor-critic
(specifically PPO,
Chapter 3) applied to
align large language
models using human
feedback as the reward
signal
─────────────────────────────────────────
This is a genuinely satisfying place to see the entire curriculum connect — reinforcement learning's conceptual backbone (built across the AI Notes and this module) turns out to be exactly the same machinery underlying some of the most prominent modern AI systems, once combined with the Deep Learning Notes' neural network toolkit.
7. Summary & Next Steps
Key Takeaways
- Tabular Q-learning (AI Notes) and this module's linear actor/critic representations both hit a scaling wall for complex, high-dimensional state spaces like raw pixels.
- Deep reinforcement learning replaces the table or linear function with a neural network, while keeping the underlying RL algorithm's logic (Q-learning's update rule, REINFORCE's gradient direction, actor-critic's TD error) unchanged.
- DQN required experience replay and target networks specifically to make neural-network-based Q-learning stable, addressing correlated experiences and a moving training target.
- This module deliberately stops at the conceptual bridge — the Deep Learning Notes provide the neural network mechanics needed to actually build and train these systems.
Module 8 Complete — Next Module
You now understand reinforcement learning's full conceptual arc: value-based methods (AI Notes), policy-based and actor-critic methods (this module), and how both scale up via neural networks into modern Deep RL and RLHF. Module 9 shifts to a very different, equally essential concern: getting any of this curriculum's models — regression, classification, clustering, or RL — safely and reliably into production.
Concept Check
- What specifically changes between tabular Q-learning, this module's linear methods, and Deep RL — and what stays exactly the same?
- Why did DQN need experience replay and target networks specifically, beyond just swapping in a neural network?
- How does PPO (Chapter 3) connect to how modern LLMs are trained via RLHF?
Next Chapter
→ Module 9: ML in Production & Capstone
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index