AI Ethics Safety And Society
Explainable AI (XAI)
Module 3, Chapter 6's expert systems had a genuine strength: every conclusion came with an explicit chain of rules that fired, fully explainable to a human. Mod
Jr Codex AI Notes
Level: Intermediate–Advanced Prerequisites: Chapter 1 Time to complete: ~25 minutes
Table of Contents
- The Black Box Problem
- Interpretable by Design vs Post-Hoc Explanation
- Feature Importance
- LIME — Explaining One Prediction at a Time
- SHAP Values (Conceptual Overview)
- When Explainability Is Legally/Ethically Required
- The Accuracy-Interpretability Trade-off
- Summary & Next Steps
1. The Black Box Problem
Module 3, Chapter 6's expert systems had a genuine strength: every conclusion came with an explicit chain of rules that fired, fully explainable to a human. Modern learned models (especially deep neural networks, covered in the Deep Learning Notes) often lack this — they can be highly accurate while being nearly impossible for a human to understand why they produced a specific output.
Symbolic AI (Module 3) Learned Models (ML/DL/NLP Notes)
───────────────────────────── ─────────────────────────────
"Loan denied BECAUSE: "Loan denied — the model
income < threshold AND said so, based on millions
debt_ratio > threshold" of learned numerical weights
that don't map to any
Fully TRANSPARENT, human-readable simple human-readable rule"
reasoning chain (Module 3, Ch.4's
inference) Often a genuine BLACK BOX
───────────────────────────── ─────────────────────────────
Explainable AI (XAI) is the field of techniques for understanding and communicating why a model produced a specific output — bridging the gap between modern learned models' accuracy and symbolic AI's transparency.
2. Interpretable by Design vs Post-Hoc Explanation
Two Fundamentally Different Strategies
─────────────────────────────────────────
Interpretable by Design: use a model TYPE that's inherently
understandable (linear regression,
decision trees — covered in the
Machine Learning Notes) — you can
read off exactly how each input
contributes to the output
Post-Hoc Explanation: use a complex, possibly opaque
model (deep neural networks),
then apply a SEPARATE technique
AFTERWARD to approximate or
probe why it made a specific
decision (Sections 4-5)
─────────────────────────────────────────
# Interpretable by design: a linear model's coefficients ARE the explanation
# prediction = w1*income + w2*debt_ratio + w3*credit_score + bias
weights = {"income": 0.4, "debt_ratio": -0.6, "credit_score": 0.8}
def explain_linear_prediction(feature_values, weights):
contributions = {feature: weights[feature] * value for feature, value in feature_values.items()}
return contributions
contributions = explain_linear_prediction(
{"income": 50000, "debt_ratio": 0.3, "credit_score": 720}, weights
)
print(contributions) # directly shows each feature's exact CONTRIBUTION to the predictionThe trade-off (fully explored in Section 7): interpretable-by-design models are often somewhat less accurate on complex problems than opaque models like deep neural networks — the choice between them is a genuine trade-off, not a strictly dominant option either way.
3. Feature Importance
A common, model-agnostic technique: measure how much each input feature actually influences a model's overall predictions, aggregated across many examples.
import random
def permutation_feature_importance(model_predict_fn, X, y, feature_names):
"""
A simplified permutation importance: shuffle ONE feature's values
at a time, measure how much accuracy DROPS — a bigger drop means
that feature was more important to the model's predictions.
"""
baseline_accuracy = compute_accuracy(model_predict_fn, X, y)
importances = {}
for i, feature in enumerate(feature_names):
X_shuffled = [row[:] for row in X]
values = [row[i] for row in X_shuffled]
random.shuffle(values)
for row, shuffled_value in zip(X_shuffled, values):
row[i] = shuffled_value
shuffled_accuracy = compute_accuracy(model_predict_fn, X_shuffled, y)
importances[feature] = baseline_accuracy - shuffled_accuracy # bigger drop = MORE important
return importances
def compute_accuracy(model_predict_fn, X, y):
predictions = [model_predict_fn(row) for row in X]
return sum(p == actual for p, actual in zip(predictions, y)) / len(y)Reading Feature Importance
─────────────────────────────────────────
If shuffling "credit_score" drops accuracy from 90% to 65%,
but shuffling "zip_code" only drops it to 88%, "credit_score"
is FAR more important to this model's decisions than "zip_code."
This works for ANY model type ("model-agnostic") since it only
needs the model's PREDICTIONS, not its internal structure.
─────────────────────────────────────────
A crucial limitation: feature importance tells you which features matter overall, across many predictions — it does NOT explain any single specific prediction, which is exactly what Section 4's LIME is designed for.
4. LIME — Explaining One Prediction at a Time
LIME (Local Interpretable Model-agnostic Explanations) explains one individual prediction by approximating the complex model's behavior with a simple, interpretable model, but only in the local neighborhood around that specific prediction.
LIME's Core Idea
─────────────────────────────────────────
The COMPLEX model might be wildly non-linear globally —
but ZOOMED IN on a tiny region around ONE specific
prediction, its behavior often looks approximately LINEAR.
LIME: 1. Generate many slightly-perturbed versions of the
instance being explained
2. Get the COMPLEX model's predictions for all of them
3. Fit a SIMPLE, interpretable model (like linear
regression) to just THIS LOCAL neighborhood of
perturbed examples
4. The simple model's coefficients become the
EXPLANATION for this one specific prediction
─────────────────────────────────────────
def simplified_lime_explain(complex_model_predict, instance, feature_names, n_samples=100):
"""A conceptual, simplified sketch of LIME's approach."""
perturbed_samples = []
predictions = []
for _ in range(n_samples):
perturbed = [value + random.gauss(0, 0.1) * value for value in instance] # small random noise
perturbed_samples.append(perturbed)
predictions.append(complex_model_predict(perturbed))
# Fit a SIMPLE (e.g. linear) model to these local samples and predictions —
# in a real LIME implementation, this uses weighted regression, weighting
# samples CLOSER to the original instance more heavily
local_explanation = fit_simple_linear_model(perturbed_samples, predictions)
return dict(zip(feature_names, local_explanation))LIME's value proposition: it works for any underlying model (truly model-agnostic, like Section 3's feature importance) but gives a specific, individual explanation ("for THIS applicant, income mattered most, credit score second") rather than only an aggregate, overall importance ranking.
5. SHAP Values (Conceptual Overview)
SHAP (SHapley Additive exPlanations) is a more mathematically rigorous alternative to LIME, grounded in cooperative game theory (a close relative of Module 5, Chapter 2's game theory) — it fairly attributes a prediction's outcome among all the contributing features.
The Shapley Value Idea (from Cooperative Game Theory)
─────────────────────────────────────────
Imagine each FEATURE is a "player" in a game, and the
prediction is the "payout" the team achieves together.
A feature's SHAP value = its AVERAGE marginal contribution,
computed across every possible ORDER in which features
could be "added" to the prediction — a fair, mathematically
principled way to split credit among features that
interact with each other.
─────────────────────────────────────────
Why SHAP Is Considered More Rigorous Than LIME
─────────────────────────────────────────
LIME: approximates locally with a simple model — fast,
intuitive, but the approximation quality can vary
and isn't mathematically guaranteed to be "fair"
SHAP: guarantees certain fairness properties (each
feature's contribution sums EXACTLY to the
difference between the prediction and a
baseline) — grounded in a formal mathematical
framework, at a higher computational cost
─────────────────────────────────────────
In practice, both LIME and SHAP are widely used, well-supported open-source tools (not something you'd typically implement from scratch) — understanding the conceptual difference (local linear approximation vs. game-theoretic fair attribution) is more valuable for a practitioner than memorizing their exact internal mathematics.
6. When Explainability Is Legally/Ethically Required
Contexts Where "It Works" Isn't Enough
─────────────────────────────────────────
Credit/lending decisions: many jurisdictions legally
require lenders to provide
SPECIFIC REASONS for a denial
(directly connecting to
Chapter 1's fairness discussion)
Medical diagnosis support: a doctor needs to understand
WHY a system flagged a
condition, to trust and
verify it appropriately
(not simply defer to it blindly)
Criminal justice risk scoring: high-stakes, individual-
liberty-affecting decisions
typically demand transparency
and the ability to CONTEST
a specific decision
Hiring decisions: increasingly regulated
(see Chapter 5) to require
some explanation of
automated decisions
─────────────────────────────────────────
The common thread: the higher the stakes for an individual (financial, medical, legal, employment), the stronger the ethical and often legal case for explainability — this is a recurring theme that connects directly into Chapter 5's regulatory landscape.
7. The Accuracy-Interpretability Trade-off
The Trade-off, Honestly Stated
─────────────────────────────────────────
Simple, interpretable models Complex, opaque models
(linear regression, small (deep neural networks,
decision trees) large ensembles)
──────────────────────────── ────────────────────────────
Easy to explain directly Often more accurate on
(Section 2) complex, high-dimensional
problems
May UNDERFIT complex, Requires post-hoc
nonlinear real-world patterns techniques (LIME/SHAP)
to explain at all —
and even then, only
approximately
─────────────────────────────────────────
This is not always a strict trade-off — sometimes a well-tuned interpretable model performs nearly as well as a complex one for a given problem, and it's a genuine best practice (covered further in the Machine Learning Notes) to always try the simplest, most interpretable model first, only reaching for opacity when it delivers a meaningful accuracy improvement that justifies the interpretability cost.
8. Summary & Next Steps
Key Takeaways
- Explainable AI (XAI) addresses the "black box problem" — modern learned models can be highly accurate while being nearly impossible for a human to directly understand, unlike Module 3's fully transparent symbolic reasoning.
- Interpretable-by-design models (linear regression, small decision trees) can be read directly; post-hoc techniques (LIME, SHAP) approximate or attribute explanations for otherwise opaque models.
- Feature importance shows which features matter overall across many predictions; LIME and SHAP explain individual, specific predictions.
- Explainability becomes a legal and ethical requirement, not just a nice-to-have, in high-stakes domains like lending, medicine, criminal justice, and hiring.
- The accuracy-interpretability trade-off is real but not absolute — always worth checking whether a simpler, interpretable model performs adequately before reaching for an opaque one.
Concept Check
- Why couldn't Module 3's expert systems have a "black box problem" the way modern deep learning models do?
- What's the key difference between what feature importance tells you and what LIME tells you?
- Name two real-world domains where explainability is often a legal or ethical requirement, and explain why.
Next Chapter
→ Chapter 3: Privacy & Security in AI Systems
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index