Reinforcement Learning For ML Practitioners
Actor-Critic Methods
Chapter 1 introduced actor-critic methods as a hybrid; Chapter 2 ended with baselines as the natural bridge. This chapter makes that bridge concrete: actor-crit
Jr Codex ML Notes
Level: Advanced Prerequisites: Chapter 2: Policy Gradient Methods Time to complete: ~25 minutes
Table of Contents
- Combining Both Worlds
- The Actor and the Critic
- How the Critic Is Trained
- How the Actor Uses the Critic
- A Complete Actor-Critic Loop
- Why This Reduces Variance Compared to REINFORCE
- A Note on Advanced Variants
- Summary & Next Steps
1. Combining Both Worlds
Chapter 1 introduced actor-critic methods as a hybrid; Chapter 2 ended with baselines as the natural bridge. This chapter makes that bridge concrete: actor-critic methods pair a policy (the "actor," Chapter 2) with a learned value function (the "critic," directly related to the AI Notes' Q-learning) that estimates how good the actor's choices actually are.
The Hybrid Structure
─────────────────────────────────────────
Actor: the POLICY (Chapter 2) — decides WHICH action to
take, given a state
Critic: a LEARNED VALUE FUNCTION (closely related to
the AI Notes' Q-learning, Module 5, Ch.5) —
evaluates HOW GOOD the actor's chosen action
turned out to be, providing a LOWER-VARIANCE
signal than Chapter 2's full-episode returns
─────────────────────────────────────────
2. The Actor and the Critic
import numpy as np
class Actor:
"""The POLICY — decides actions. Directly Chapter 2's softmax policy."""
def __init__(self, n_features, n_actions):
self.weights = np.random.randn(n_features, n_actions) * 0.01
def get_action_probabilities(self, state):
scores = state @ self.weights
exp_scores = np.exp(scores - np.max(scores))
return exp_scores / np.sum(exp_scores)
def sample_action(self, state):
probs = self.get_action_probabilities(state)
return np.random.choice(len(probs), p=probs)
class Critic:
"""The VALUE FUNCTION — estimates V(s), the expected return from state s.
Directly related to the AI Notes' Module 5, Ch.4 value function concept."""
def __init__(self, n_features):
self.weights = np.random.randn(n_features) * 0.01
def get_value(self, state):
return state @ self.weights3. How the Critic Is Trained
The critic learns to predict expected return, using a Temporal Difference (TD) update — directly related to the AI Notes' Q-learning update rule (Module 5, Chapter 5), applied to state values rather than state-action values.
def critic_update(critic, state, reward, next_state, done, learning_rate=0.01, gamma=0.99):
"""
Directly parallels the AI Notes' Q-learning update:
Q(s,a) ← Q(s,a) + α[r + γ·max_a'Q(s',a') − Q(s,a)]
Here, for the CRITIC's value function:
V(s) ← V(s) + α[r + γ·V(s') − V(s)]
"""
current_value = critic.get_value(state)
next_value = 0 if done else critic.get_value(next_state)
td_target = reward + gamma * next_value
td_error = td_target - current_value # the SAME "TD error" idea from the AI Notes
critic.weights += learning_rate * td_error * state # gradient step toward the TD target
return td_error # this becomes the ACTOR's training signal (Section 4)Directly Connects to the AI Notes' Q-Learning Update Rule
─────────────────────────────────────────
AI Notes (Module 5, Ch.5): Q(s,a) ← Q(s,a) + α[r + γ·max Q(s',a') − Q(s,a)]
This chapter's critic: V(s) ← V(s) + α[r + γ·V(s') − V(s)]
IDENTICAL structure — an "old estimate," a "target" built from
the observed reward plus the (discounted) NEXT state's value,
and a step toward closing the gap. The critic IS essentially
performing Q-learning's core update mechanism, just for STATE
values instead of state-ACTION values.
─────────────────────────────────────────
4. How the Actor Uses the Critic
The TD error computed by the critic (Section 3) serves as the actor's advantage estimate — directly replacing Chapter 2's noisy, full-episode return with a much lower-variance, step-by-step signal.
def actor_update(actor, state, action, td_error, learning_rate=0.01):
"""Uses the CRITIC's TD error as the advantage — replacing
Chapter 2's noisy full-episode G with a per-STEP estimate."""
probabilities = actor.get_action_probabilities(state)
grad_log_prob = -probabilities.copy()
grad_log_prob[action] += 1
actor.weights += learning_rate * td_error * np.outer(state, grad_log_prob)Chapter 2's REINFORCE vs This Chapter's Actor-Critic
─────────────────────────────────────────
REINFORCE (Ch.2): waits for the ENTIRE episode to finish,
uses the TOTAL discounted return —
high variance, but unbiased
Actor-Critic (here): updates AFTER EVERY SINGLE STEP,
using the critic's IMMEDIATE TD
error as a lower-variance (though
slightly biased, since the critic
itself is still learning and
imperfect) advantage estimate
─────────────────────────────────────────
5. A Complete Actor-Critic Loop
def train_actor_critic(actor, critic, environment, n_episodes=1000, gamma=0.99):
for episode in range(n_episodes):
state = environment.reset()
done = False
while not done:
action = actor.sample_action(state)
next_state, reward, done = environment.step(action)
# Critic updates FIRST, producing the TD error
td_error = critic_update(critic, state, reward, next_state, done, gamma=gamma)
# Actor uses that SAME TD error as its advantage signal
actor_update(actor, state, action, td_error)
state = next_stateThe Full Actor-Critic Cycle, Per Step
─────────────────────────────────────────
1. ACTOR chooses an action, given the current state
2. Environment responds with a reward and next state
3. CRITIC computes the TD error (how surprising was this
outcome, relative to what the critic expected?)
4. ACTOR updates using the critic's TD error as its advantage
5. CRITIC also updates its own value estimate toward the
observed TD target
6. Repeat, one step at a time — no need to wait for a full
episode, unlike Chapter 2's REINFORCE
─────────────────────────────────────────
6. Why This Reduces Variance Compared to REINFORCE
The Core Trade-off
─────────────────────────────────────────
REINFORCE (Ch.2): UNBIASED (the true expected gradient),
but HIGH VARIANCE (depends on entire,
often noisy episode outcomes)
Actor-Critic (here): LOWER VARIANCE (updates use immediate,
per-step TD errors, not noisy full
returns), but SLIGHTLY BIASED
(since the critic's own value
estimates are imperfect, especially
early in training)
─────────────────────────────────────────
This is precisely Module 1, Chapter 4's bias-variance tradeoff, applied to gradient estimation itself — actor-critic accepts a small amount of bias (an imperfect critic) in exchange for a substantial reduction in variance, generally leading to faster, more stable training than pure REINFORCE.
7. A Note on Advanced Variants
This chapter covers the foundational actor-critic idea; production RL systems typically use more sophisticated variants building on the same core structure:
Names Worth Recognizing (Beyond This Introductory Scope)
─────────────────────────────────────────
A2C / A3C (Advantage Actor-Critic): formalizes the
"advantage" signal
more carefully than
the raw TD error used here
PPO (Proximal Policy Optimization): adds a constraint
preventing the
actor from changing
TOO drastically in
one update — widely
used in modern
large-scale RL,
including RLHF
(Reinforcement
Learning from Human
Feedback) for
training LLMs
(NLP/LLM Notes)
DDPG / SAC: actor-critic
variants
specifically
designed for
CONTINUOUS
action spaces
(Chapter 1,
Section 5's
motivation)
─────────────────────────────────────────
PPO's connection to LLM training is worth highlighting directly: the RLHF technique used to align modern large language models (mentioned in the AI Notes' alignment chapter, covered fully in the NLP/LLM Notes) uses exactly this actor-critic structure — a language model "actor" generating text, and a learned "critic"/reward model evaluating how good that text is, refined via PPO.
8. Summary & Next Steps
Key Takeaways
- Actor-critic methods pair a policy (the actor, Chapter 2) with a learned value function (the critic), directly related to the AI Notes' Q-learning update mechanism but applied to state values.
- The critic's TD error serves as the actor's advantage signal, replacing Chapter 2's noisy full-episode return with a lower-variance, per-step estimate.
- This trades a small amount of bias (from the critic's own imperfect, still-learning estimates) for substantially reduced variance compared to pure REINFORCE — a direct instance of the bias-variance tradeoff applied to gradient estimation.
- Modern variants (A2C, PPO, DDPG, SAC) build on this same actor-critic structure — PPO in particular is the technique behind RLHF, used to align modern large language models.
Module 8 Complete (Theory) — Next: Scaling Up
You now understand both major reinforcement learning families: value-based (from the AI Notes) and policy-based/actor-critic (this module). Chapter 4 addresses the practical scaling problem — why tabular methods break down for complex, real-world environments, and how neural networks solve it.
Concept Check
- What specific role does the critic play in an actor-critic method, and how does it relate to the AI Notes' Q-learning?
- Why does actor-critic training generally have lower variance than pure REINFORCE, and what's the cost of that improvement?
- How does PPO's actor-critic structure connect to how modern LLMs are aligned via RLHF?
Next Chapter
→ Chapter 4: From Tabular RL to Deep RL
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index