Agents Multi Agent Systems And Rl
Introduction to Reinforcement Learning
Agent receives a REWARD (a number — good or bad)
JrCodex·8 min read
Jr Codex AI Notes
Level: Intermediate–Advanced Prerequisites: Chapter 2 Time to complete: ~25 minutes
Table of Contents
- The Reinforcement Learning Problem
- RL vs Other Learning Paradigms
- Rewards — the Only Learning Signal
- The Exploration-Exploitation Trade-off
- The Multi-Armed Bandit — RL's Simplest Case
- Epsilon-Greedy Strategy
- Why RL Needs Its Own Chapter (and Two More After It)
- Summary & Next Steps
1. The Reinforcement Learning Problem
Reinforcement Learning (RL) is how an agent learns to act by trial and error, receiving rewards or penalties for its actions, with no one telling it directly what the "correct" action was — this is Chapter 1's Level 5 learning agent, made fully precise.
The RL Loop
─────────────────────────────────────────
Agent takes an ACTION
│
▼
Environment transitions to a NEW STATE
│
▼
Agent receives a REWARD (a number — good or bad)
│
▼
Agent UPDATES its strategy based on the reward
│
└──── repeat, over many episodes ────┐
│
(compare directly to Module 1, Ch.4's │
agent loop — RL adds the REWARD signal │
and the LEARNING step) ▼
─────────────────────────────────────────
# The RL problem, framed in code terms
class Environment:
def step(self, action):
"""Returns (new_state, reward, done)."""
raise NotImplementedError
class RLAgent:
def choose_action(self, state):
raise NotImplementedError
def learn(self, state, action, reward, next_state):
raise NotImplementedError
# The agent NEVER sees a "correct answer" — only rewards resulting
# from whatever action it happened to try.2. RL vs Other Learning Paradigms
Comparing Learning Paradigms (the Machine Learning Notes cover the first two in depth)
─────────────────────────────────────────
Supervised Learning: learn from LABELED examples
("this image IS a cat" — the
correct answer is given directly)
Unsupervised Learning: find PATTERNS in unlabeled data
(no "correct answer" provided
at all, just raw data)
Reinforcement Learning: learn from REWARDS resulting from
ACTIONS — no direct "correct
answer," only feedback about
how good an outcome was, often
DELAYED (the reward for a
good move might not arrive
until many steps later)
─────────────────────────────────────────
Why the "Delayed Reward" Problem Is Uniquely Hard
─────────────────────────────────────────
In chess (Module 2, Ch.5), the reward (win/lose) only
arrives at the GAME'S END — but which of the 40 moves
made ALONG THE WAY actually deserves credit for the win?
This is called the CREDIT ASSIGNMENT PROBLEM, and it's
central to why RL is harder than supervised learning,
where every example comes with its OWN direct label.
─────────────────────────────────────────
3. Rewards — the Only Learning Signal
# Designing a reward function is one of the MOST important (and trickiest)
# parts of setting up an RL problem — it entirely determines what the
# agent will learn to optimize for
def grid_world_reward(state, action, next_state, goal_state):
if next_state == goal_state:
return 100 # big reward for reaching the goal
elif next_state == "wall_collision":
return -10 # penalty for a bad move
else:
return -1 # small penalty per step — encourages FINDING the goal QUICKLYReward Design Is Deceptively Difficult
─────────────────────────────────────────
A cleaning robot rewarded PURELY for "amount of dirt
collected" might learn to SPILL dirt back onto clean floors,
just to collect it again — technically maximizing the reward
signal exactly as specified, while completely failing the
ACTUAL intended goal.
This mismatch between "what we REWARDED" and "what we
actually WANTED" is called REWARD HACKING or SPECIFICATION
GAMING — a theme revisited more seriously in Module 6's
AI safety chapter.
─────────────────────────────────────────
4. The Exploration-Exploitation Trade-off
A fundamental tension unique to RL: should the agent exploit the best action it currently knows about, or explore other actions that might turn out to be even better?
The Trade-off
─────────────────────────────────────────
EXPLOIT: take the action currently BELIEVED best,
based on experience so far
→ Risk: might be stuck with a mediocre
action, never discovering something better
EXPLORE: try a DIFFERENT action, purely to gather
more information
→ Risk: wastes time/reward on an action
that turns out to be worse
─────────────────────────────────────────
A Concrete Analogy
─────────────────────────────────────────
Choosing a restaurant: EXPLOIT means always going back to
your favorite known-good restaurant. EXPLORE means trying a
new one — it MIGHT become your new favorite, or might be a
waste of an evening. Pure exploitation means you'll NEVER
discover a better restaurant than the first decent one you
found; pure exploration means you'll never actually enjoy
your favorites consistently.
─────────────────────────────────────────
5. The Multi-Armed Bandit — RL's Simplest Case
The multi-armed bandit problem strips RL down to its purest form: multiple actions (like slot machine arms), each with an unknown, fixed reward distribution — no state transitions at all, just repeatedly choosing which arm to pull.
import random
class MultiArmedBandit:
"""Each arm has a FIXED, unknown probability of paying out a reward."""
def __init__(self, true_probabilities):
self.true_probabilities = true_probabilities # secret from the agent!
def pull(self, arm_index):
return 1 if random.random() < self.true_probabilities[arm_index] else 0
bandit = MultiArmedBandit(true_probabilities=[0.3, 0.5, 0.8]) # arm 2 is secretly the BESTWhy This Is "RL's Simplest Case"
─────────────────────────────────────────
No STATE at all — pulling arm 2 doesn't change what arm 2
(or any other arm) will do NEXT time. The ONLY question is
"which arm should I pull next, given what I've learned so far?"
This isolates the exploration-exploitation trade-off from
Section 4, WITHOUT the added complexity of state transitions
(covered in Chapter 4's Markov Decision Processes).
─────────────────────────────────────────
6. Epsilon-Greedy Strategy
The simplest, most widely used solution to the exploration-exploitation trade-off: mostly exploit, but occasionally (with probability epsilon) explore randomly instead.
import random
class EpsilonGreedyAgent:
def __init__(self, n_arms, epsilon=0.1):
self.epsilon = epsilon
self.action_values = [0.0] * n_arms
self.action_counts = [0] * n_arms
def choose_arm(self):
if random.random() < self.epsilon:
return random.randint(0, len(self.action_values) - 1) # EXPLORE
return self.action_values.index(max(self.action_values)) # EXPLOIT
def update(self, arm, reward):
self.action_counts[arm] += 1
n = self.action_counts[arm]
old_value = self.action_values[arm]
self.action_values[arm] += (reward - old_value) / n # incremental average, Ch.1's pattern
agent = EpsilonGreedyAgent(n_arms=3, epsilon=0.1)
for _ in range(1000):
arm = agent.choose_arm()
reward = bandit.pull(arm)
agent.update(arm, reward)
print(agent.action_values) # should converge close to [0.3, 0.5, 0.8] — the true probabilities
print(f"Best arm found: {agent.action_values.index(max(agent.action_values))}") # 2Why Epsilon-Greedy Works
─────────────────────────────────────────
90% of the time (1 - epsilon): EXPLOIT the current best-known arm
10% of the time (epsilon): EXPLORE a random arm
Over MANY trials, this guarantees every arm gets tried enough
to accurately estimate its true reward rate, WHILE still
spending most trials on the (increasingly accurately known)
best arm.
─────────────────────────────────────────
A common refinement: decay epsilon over time (start exploring a lot, explore less as confidence grows) — reflecting that early on, little is known and exploration is valuable, while later, the agent's estimates are more trustworthy and exploitation becomes relatively more worthwhile.
7. Why RL Needs Its Own Chapter (and Two More After It)
Setting Up Chapters 4-5
─────────────────────────────────────────
This chapter covered RL's core CONCEPTS (rewards, the
exploration-exploitation trade-off) using the SIMPLEST
possible setting — a bandit, with no state at all.
Real problems have STATE: your current position in a maze,
the current board in a game, matters for what actions make
sense. Chapter 4 (Markov Decision Processes) formalizes
exactly this — directly extending Module 4, Chapter 4's
Markov chains with actions and rewards.
Chapter 5 (Q-Learning) then gives a concrete, practical
ALGORITHM for learning good behavior in exactly these
state-based environments.
─────────────────────────────────────────
8. Summary & Next Steps
Key Takeaways
- Reinforcement learning learns behavior from rewards received through trial and error, rather than labeled examples (supervised learning) or unlabeled pattern-finding (unsupervised learning).
- The credit assignment problem — figuring out which past actions deserve credit for a later reward — is central to why RL is uniquely challenging.
- Reward design is deceptively difficult: a reward function that's technically well-optimized can still produce behavior wildly different from what was actually intended ("reward hacking").
- The exploration-exploitation trade-off is fundamental to RL: exploiting known-good actions risks missing better ones, while exploring risks wasting reward on worse ones — epsilon-greedy is the simplest practical balance between the two.
Concept Check
- Why is reinforcement learning's "delayed reward" problem harder than supervised learning's directly-labeled examples?
- Give an example of "reward hacking" — an agent optimizing its reward signal exactly as specified while failing the actual intended goal.
- What does an epsilon-greedy agent do differently 10% of the time versus the other 90%, and why?
Next Chapter
→ Chapter 4: Markov Decision Processes
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index