Agents Multi Agent Systems And Rl
Agent Types Revisited
Module 1, Chapter 4 defined the agent loop abstractly (perceive → decide → act) and the PEAS framework for specifying requirements. This chapter organizes agent
Jr Codex AI Notes
Level: Intermediate Prerequisites: Module 4: Reasoning Under Uncertainty Time to complete: ~25 minutes
Table of Contents
- From the Abstract Agent Loop to Concrete Architectures
- Level 1 — Simple Reflex Agents
- Level 2 — Model-Based Reflex Agents
- Level 3 — Goal-Based Agents
- Level 4 — Utility-Based Agents
- Level 5 — Learning Agents
- How This Curriculum Maps onto These Five Levels
- Summary & Next Steps
1. From the Abstract Agent Loop to Concrete Architectures
Module 1, Chapter 4 defined the agent loop abstractly (perceive → decide → act) and the PEAS framework for specifying requirements. This chapter organizes agent architectures — concrete internal designs — into five increasingly sophisticated levels, each solving a limitation of the one before it.
The Five Levels, At a Glance
─────────────────────────────────────────
1. Simple Reflex → reacts to CURRENT percept only
2. Model-Based Reflex → maintains an internal STATE/model
3. Goal-Based → plans toward an explicit GOAL
4. Utility-Based → weighs TRADE-OFFS between outcomes
5. Learning → improves from EXPERIENCE over time
─────────────────────────────────────────
2. Level 1 — Simple Reflex Agents
Decides purely from the current percept, using condition-action rules — no memory of the past at all.
class SimpleReflexAgent:
"""Acts based ONLY on the current percept — no memory whatsoever."""
def __init__(self, rules):
self.rules = rules # dict: {condition: action}
def act(self, percept):
return self.rules.get(percept, "no_action")
thermostat = SimpleReflexAgent({
"too_cold": "turn_on_heater",
"too_hot": "turn_on_AC",
"just_right": "do_nothing",
})
print(thermostat.act("too_cold")) # "turn_on_heater"Works Well When... Fails When...
───────────────────────────── ─────────────────────────────
Environment is fully observable State is HIDDEN or partial —
(Module 1, Ch.4) and deterministic the agent has no way to
"remember" anything it
can't currently perceive
───────────────────────────── ─────────────────────────────
3. Level 2 — Model-Based Reflex Agents
Adds an internal model of the world — letting the agent handle partially observable environments by remembering relevant history.
class ModelBasedAgent:
"""Maintains an internal world model, updated by each new percept."""
def __init__(self):
self.internal_state = {}
def update_state(self, percept):
if percept.get("obstacle_detected"):
self.internal_state["obstacle_ahead"] = True
else:
self.internal_state["obstacle_ahead"] = False
self.internal_state["battery"] = percept.get("battery_level", 100)
def decide(self):
if self.internal_state.get("battery", 100) < 10:
return "return_to_base"
if self.internal_state.get("obstacle_ahead"):
return "turn_away"
return "move_forward"
def act(self, percept):
self.update_state(percept)
return self.decide()
robot = ModelBasedAgent()
print(robot.act({"battery_level": 70, "obstacle_detected": True})) # "turn_away"
print(robot.act({"battery_level": 8, "obstacle_detected": False})) # "return_to_base"This is precisely a Markov chain's hidden-state idea (Module 4, Chapter 4) applied to agent design — the agent maintains a belief about the world's current state, updated from each new observation, rather than assuming the current percept tells the whole story.
4. Level 3 — Goal-Based Agents
Adds an explicit goal, and uses search or planning (Modules 2-3) to find action sequences that achieve it — rather than just reacting, the agent looks ahead.
class GoalBasedAgent:
"""Uses an explicit goal and a PLAN (Module 3, Ch.5) to decide actions."""
def __init__(self, goal):
self.goal = goal
self.plan = []
def formulate_plan(self, current_state, planner_fn):
"""planner_fn: a Module 3-style STRIPS planner, or Module 2-style search"""
self.plan = planner_fn(current_state, self.goal)
def act(self):
if self.plan:
return self.plan.pop(0)
return "goal_achieved"
# planner_fn would be Module 2's search or Module 3's STRIPS planning,
# reused directly here as the agent's internal decision-making engine
delivery_robot = GoalBasedAgent(goal="deliver_package_to_customer")This is where Module 2 (search) and Module 3 (planning) plug directly into the agent framework — a goal-based agent's formulate_plan step is exactly what those modules' algorithms compute.
5. Level 4 — Utility-Based Agents
Rather than a single fixed goal, weighs multiple, possibly competing outcomes using utility (Module 4, Chapter 5) — appropriate when "success" isn't binary, but a matter of degree and trade-offs.
class UtilityBasedAgent:
"""Chooses the action with the highest EXPECTED UTILITY (Module 4, Ch.5)."""
def __init__(self, utility_fn):
self.utility_fn = utility_fn
def choose_action(self, actions_with_outcomes):
best_action, best_eu = None, float("-inf")
for action, outcomes in actions_with_outcomes.items():
eu = sum(prob * self.utility_fn(outcome) for outcome, prob in outcomes.items())
if eu > best_eu:
best_action, best_eu = action, eu
return best_action, best_eu
def treatment_utility(outcome):
return {"full_recovery": 1.0, "partial_recovery": 0.6, "no_change": 0.0, "worse": -0.5}.get(outcome, 0)
medical_agent = UtilityBasedAgent(treatment_utility)
options = {
"Drug_A": {"full_recovery": 0.7, "partial_recovery": 0.2, "worse": 0.1},
"Surgery": {"full_recovery": 0.85, "worse": 0.15},
}
best, eu = medical_agent.choose_action(options)
print(f"Best choice: {best} (EU={eu:.2f})")This is Module 4, Chapter 5's Maximum Expected Utility principle, now embedded directly as an agent's decision-making architecture — a goal-based agent asks "does this achieve my goal?" (binary); a utility-based agent asks "how good is this outcome, weighed against the alternatives?" (a spectrum).
6. Level 5 — Learning Agents
Every prior level had its knowledge/rules/utility function hand-coded in advance. A learning agent improves its own performance from experience — the conceptual bridge into this module's remaining chapters (reinforcement learning, Chapters 3-5) and the entire Machine Learning notes.
Learning Agent Architecture
─────────────────────────────────────────
Environment
│ percepts
▼
┌────────────────────────────────────────┐
│ LEARNING ELEMENT ←── critic ←── performance standard
│ │ improves
│ ▼
│ PERFORMANCE ELEMENT (decision making —
│ │ any of Levels 1-4 above)
│ ▼
│ PROBLEM GENERATOR (suggests exploring
│ new actions, to learn more)
└────────────────────────────────────────┘
│ actions
▼
Environment
─────────────────────────────────────────
class SimpleLearningAgent:
"""
A minimal learning agent: tracks which actions have historically
led to good outcomes, and gradually favors them more.
(A simplified preview of Q-learning, Chapter 5.)
"""
def __init__(self, actions):
self.action_values = {action: 0.0 for action in actions}
self.action_counts = {action: 0 for action in actions}
def choose_action(self):
return max(self.action_values, key=self.action_values.get)
def learn(self, action, reward):
"""Update this action's estimated value using the observed reward."""
self.action_counts[action] += 1
n = self.action_counts[action]
old_value = self.action_values[action]
# Incrementally average in the new reward
self.action_values[action] = old_value + (reward - old_value) / n
agent = SimpleLearningAgent(actions=["left", "right", "straight"])
agent.learn("right", reward=10)
agent.learn("right", reward=8)
agent.learn("left", reward=2)
print(agent.action_values) # {'left': 2.0, 'right': 9.0, 'straight': 0.0}
print(agent.choose_action()) # "right" — learned to prefer it from experienceThis tiny example is a real, working preview of reinforcement learning — the "Performance Element" here isn't hand-coded; it emerges from observed rewards, exactly the theme Chapters 3-5 develop fully.
7. How This Curriculum Maps onto These Five Levels
Connecting Every Prior Module to This Hierarchy
─────────────────────────────────────────
Level 1-2 (Reflex): basic condition-action rules —
Module 3's expert systems (Ch.6)
are essentially sophisticated
simple-reflex agents
Level 3 (Goal-Based): Module 2 (search) and Module 3
(planning) provide the actual
planning ENGINE this level needs
Level 4 (Utility-Based): Module 4, Chapter 5's decision
theory provides the formal
utility/expected-utility machinery
Level 5 (Learning): THIS module's remaining chapters
(reinforcement learning) plus
the entire Machine Learning,
Deep Learning, and NLP/LLM notes
─────────────────────────────────────────
This mapping is a useful way to see how the entire curriculum — not just this one module — assembles into a coherent picture of "how to build increasingly sophisticated agents."
8. Summary & Next Steps
Key Takeaways
- Agent architectures progress through five levels of sophistication: simple reflex (no memory), model-based reflex (internal state), goal-based (plans toward a goal), utility-based (weighs trade-offs), and learning (improves from experience).
- Each level solves a specific limitation of the previous one — reflex agents can't handle partial observability, goal-based agents can't handle graded trade-offs, and none of the first four levels improve their own knowledge over time.
- Goal-based and utility-based agents directly reuse this curriculum's earlier modules: Module 2/3's search and planning for goals, Module 4's decision theory for utility.
- A learning agent's performance element emerges from observed experience/rewards rather than being hand-coded — the conceptual foundation for reinforcement learning, covered in this module's remaining chapters.
Concept Check
- Why can't a simple reflex agent handle a partially observable environment well, while a model-based reflex agent can?
- What's the key difference between a goal-based agent and a utility-based agent?
- What makes a learning agent structurally different from the first four levels, even though it can still use any of them as its "performance element"?
Next Chapter
→ Chapter 2: Multi-Agent Systems & Game Theory
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index