Reinforcement Learning For ML Practitioners
RL Recap & Where It Fits in ML
Module 1, Chapter 2 introduced reinforcement learning as the third paradigm and explicitly deferred its foundations to the AI Notes' Module 5 — which covers age
Jr Codex ML Notes
Level: Intermediate Prerequisites: AI Notes, Module 5: Agents, Multi-Agent Systems & RL Time to complete: ~20 minutes
Table of Contents
- Why This Chapter Is Short
- The Foundation You Already Have
- Value-Based Methods — What You Already Know
- Policy-Based Methods — What's New in This Module
- Why Policy-Based Methods Are Needed At All
- Where RL Sits in the Three-Paradigm Map
- Summary & Next Steps
1. Why This Chapter Is Short
Module 1, Chapter 2 introduced reinforcement learning as the third paradigm and explicitly deferred its foundations to the AI Notes' Module 5 — which covers agent types, multi-agent systems, the RL problem, Markov Decision Processes, and Q-learning in complete depth. This chapter is intentionally brief: a recap connecting that foundation to what's genuinely new in this module (Chapters 2-4), not a re-derivation.
2. The Foundation You Already Have
What the AI Notes' Module 5 Already Covered
─────────────────────────────────────────
Ch.1: Agent Types Revisited — reflex, model-based,
goal-based, utility-based,
learning agents
Ch.2: Multi-Agent Systems & game theory, Nash
Game Theory equilibrium
Ch.3: Introduction to the RL problem, rewards,
Reinforcement Learning exploration-exploitation,
epsilon-greedy, the
multi-armed bandit
Ch.4: Markov Decision states, actions, the
Processes Bellman equation,
value iteration
Ch.5: Q-Learning Fundamentals Q-values, the
Q-learning update
rule, model-free
learning
─────────────────────────────────────────
If any of this feels unfamiliar, stop here and complete that module first — everything in this chapter's remaining sections and Chapters 2-4 builds directly on it.
3. Value-Based Methods — What You Already Know
Q-learning (AI Notes, Module 5, Chapter 5) is a value-based method — it learns the value of taking each action in each state (Q(s,a)), then derives behavior indirectly: "in this state, take whichever action has the highest learned Q-value."
# Directly from the AI Notes — the core value-based idea, recapped
def best_action(q_table, state, actions):
"""The POLICY is DERIVED from the learned VALUES, not learned directly."""
return max(actions, key=lambda a: q_table.get((state, a), 0.0))The Value-Based Recipe
─────────────────────────────────────────
1. Learn Q(s,a) for every state-action pair
2. DERIVE the policy: always pick argmax_a Q(s,a)
The POLICY is a byproduct of the learned VALUES — never
learned or represented directly, on its own.
─────────────────────────────────────────
4. Policy-Based Methods — What's New in This Module
Chapters 2-3 introduce a fundamentally different family: policy-based methods learn the policy — the mapping from state to action — directly, without ever explicitly computing Q-values for every state-action pair first.
Value-Based (AI Notes) Policy-Based (Chapters 2-3)
─────────────────────────────────────────────────────────────────
Learn Q(s,a) for EVERY state- Learn π(a|s) DIRECTLY —
action pair, DERIVE the policy a probability
by taking the argmax distribution over
actions, given a state
Naturally suited to DISCRETE Naturally handles
action spaces (a finite Q-table CONTINUOUS action
entry per state-action pair) spaces (e.g. "how
much force to
apply," not just
"left or right")
─────────────────────────────────────────────────────────────────
5. Why Policy-Based Methods Are Needed At All
A Concrete Limitation of Value-Based Methods
─────────────────────────────────────────
Q-learning (AI Notes, Ch.5) requires computing max_a Q(s,a) —
finding the BEST action among all possible ones.
For a DISCRETE action space (left/right/up/down), this is
trivial — just check every option.
For a CONTINUOUS action space (e.g. "steering angle between
-30° and +30°"), there are INFINITELY many possible actions
to maximize over — checking every one is impossible.
Policy-based methods sidestep this ENTIRELY by learning a
DIRECT mapping from state to action (or a probability
distribution over actions), never needing to search over an
infinite space for a maximum.
─────────────────────────────────────────
This is precisely why robotics and continuous-control problems (steering a car, controlling a robot arm's joint angles) rely heavily on policy-based and actor-critic methods (Chapter 3), while discrete, game-like problems (AI Notes' grid-world example) work well with pure Q-learning.
6. Where RL Sits in the Three-Paradigm Map
Directly reprising Module 1, Chapter 2's map — worth restating with this module's added detail:
Reinforcement Learning's Internal Structure
─────────────────────────────────────────
Reinforcement Learning
/ \
Value-Based Policy-Based
(AI Notes, Module 5) (Chapters 2-3, this module)
│ │
Q-Learning Policy Gradients (Ch.2)
│
Actor-Critic (Ch.3)
(COMBINES both approaches)
─────────────────────────────────────────
Actor-Critic methods (Chapter 3) aren't a third, separate category — they're a genuine hybrid, using a value-based component (the "critic," closely related to AI Notes' Q-learning) to help train a policy-based component (the "actor") more efficiently than pure policy gradients alone.
7. Summary & Next Steps
Key Takeaways
- This module builds directly on the AI Notes' Module 5, which covered the RL problem, MDPs, and Q-learning (a value-based method) in full depth.
- Value-based methods learn Q(s,a) for every state-action pair and derive the policy by taking the action with the highest value; policy-based methods (this module's new content) learn the policy directly.
- Policy-based methods handle continuous action spaces naturally, since they never need to search over infinitely many actions for a maximum — a genuine limitation of pure value-based methods like Q-learning.
- Actor-critic methods (Chapter 3) combine both approaches, using a value-based "critic" to help train a policy-based "actor" more efficiently.
Concept Check
- What's the key structural difference between how value-based and policy-based methods represent an agent's behavior?
- Why do continuous action spaces pose a genuine problem for pure Q-learning?
- In what sense are actor-critic methods a "hybrid" rather than a third, separate category?
Next Chapter
→ Chapter 2: Policy Gradient Methods
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index