Machine Learning

ML In Production And Capstone

Model Monitoring, Drift & Retraining

Every algorithm in Modules 3-6 was trained once, on a fixed dataset, and evaluated (Module 1, Chapter 5; Module 4, Chapter 6) as if that were the end of the sto

JrCodex·8 min read

Jr Codex ML Notes

Level: Advanced Prerequisites: Chapter 2: Model Deployment Basics Time to complete: ~25 minutes

Table of Contents

  1. A Deployed Model Is Never "Done"
  2. What Can Go Wrong After Deployment
  3. Data Drift
  4. Concept Drift
  5. Detecting Drift Statistically
  6. Monitoring Prediction Distributions
  7. When and How to Retrain
  8. Building a Monitoring Dashboard
  9. Summary & Next Steps

1. A Deployed Model Is Never "Done"

Every algorithm in Modules 3-6 was trained once, on a fixed dataset, and evaluated (Module 1, Chapter 5; Module 4, Chapter 6) as if that were the end of the story. In production, a model keeps making predictions on new, real-world data indefinitely — and the real world doesn't stay still. This chapter covers what happens after Chapter 2's deployment, often the least glamorous but most operationally important part of the ML workflow (Module 1, Chapter 3).

2. What Can Go Wrong After Deployment

Failure Modes Specific to a LIVE, Deployed Model
─────────────────────────────────────────
  Data drift (Section 3):        the INPUT data's distribution
                                    changes over time
  Concept drift (Section 4):        the RELATIONSHIP between
                                       inputs and the target
                                       changes over time
  Data pipeline bugs:                  upstream data sources
                                          change format, break,
                                          or introduce errors
  Feedback loops:                        the model's OWN
                                            predictions influence
                                            future data (e.g. a
                                            recommendation
                                            system shapes what
                                            users see, which
                                            shapes future
                                            training data)
─────────────────────────────────────────

3. Data Drift

Data drift occurs when the distribution of input features seen in production diverges from what the model was trained on — directly connects to the Data Science Notes' distribution/EDA concepts, now monitored continuously rather than checked once.

import numpy as np
from scipy import stats
 
# Training data's age distribution (Module 1, Ch.1's descriptive stats)
training_ages = np.random.normal(35, 10, 1000)
 
# PRODUCTION data collected a few months later — the population has SHIFTED
production_ages_month1 = np.random.normal(35, 10, 1000)      # similar — no drift yet
production_ages_month6 = np.random.normal(48, 12, 1000)      # DIFFERENT — genuine drift
 
# Kolmogorov-Smirnov test — checks if two samples come from the SAME distribution
stat, p_value_month1 = stats.ks_2samp(training_ages, production_ages_month1)
stat, p_value_month6 = stats.ks_2samp(training_ages, production_ages_month6)
 
print(f"Month 1 vs training — p-value: {p_value_month1:.4f}")      # likely high — NO significant drift
print(f"Month 6 vs training — p-value: {p_value_month6:.4f}")      # likely very low — SIGNIFICANT drift detected
Why Data Drift Matters Even Without Retraining Yet
─────────────────────────────────────────
  A model's LEARNED patterns (Modules 3-6) reflect the
  TRAINING data's distribution. If the REAL WORLD'S input
  distribution shifts substantially away from that, the
  model is now making predictions in a REGION it has less
  (or no) genuine experience with — performance can degrade
  silently, well BEFORE anyone notices without active monitoring.
─────────────────────────────────────────

4. Concept Drift

A subtler, often more serious problem: even if input data looks the same, the actual relationship between inputs and the target can change.

A Concrete Example
─────────────────────────────────────────
  A fraud detection model (Module 2, Ch.5) learned that
  "large purchase + new account + foreign IP address" strongly
  predicts fraud, based on HISTORICAL fraud patterns.

  Fraudsters ADAPT — six months later, they've changed
  tactics entirely, and the OLD pattern no longer predicts
  fraud reliably, while a NEW, different pattern now does.

  The INPUT data (transaction records) still LOOKS structurally
  similar — but the underlying RELATIONSHIP between features
  and the "is fraud" label has fundamentally SHIFTED.
─────────────────────────────────────────

Data Drift vs Concept Drift — the Key Distinction
─────────────────────────────────────────
  Data drift:      P(X) changes — the INPUT features' own
                     distribution shifts

  Concept drift:      P(y | X) changes — the RELATIONSHIP
                         between inputs and target shifts,
                         even if inputs LOOK similar

  Concept drift is generally HARDER to detect, since it
  requires comparing PREDICTIONS against ACTUAL OUTCOMES
  (ground truth), which often arrive LATE or not at all in
  production (e.g. "was this actually fraud?" might only be
  confirmed weeks later, if ever)
─────────────────────────────────────────

5. Detecting Drift Statistically

Beyond the KS-test example in Section 3, several complementary techniques apply — directly reusing the Data Science Notes' statistical toolkit.

import numpy as np
 
def population_stability_index(expected, actual, bins=10):
    """
    PSI — a standard industry metric for measuring distribution
    shift, commonly used in credit risk and fraud modeling.
    """
    breakpoints = np.percentile(expected, np.linspace(0, 100, bins + 1))
    breakpoints[0], breakpoints[-1] = -np.inf, np.inf
 
    expected_pct = np.histogram(expected, breakpoints)[0] / len(expected)
    actual_pct = np.histogram(actual, breakpoints)[0] / len(actual)
 
    expected_pct = np.clip(expected_pct, 1e-6, None)      # avoid division by zero
    actual_pct = np.clip(actual_pct, 1e-6, None)
 
    psi = np.sum((actual_pct - expected_pct) * np.log(actual_pct / expected_pct))
    return psi
 
psi_value = population_stability_index(training_ages, production_ages_month6)
print(f"PSI: {psi_value:.4f}")
Interpreting PSI (a Common Industry Convention)
─────────────────────────────────────────
  PSI < 0.1:      no significant population shift
  PSI 0.1 - 0.25:    moderate shift — worth investigating
  PSI > 0.25:           significant shift — likely requires action
                          (Section 7's retraining decision)
─────────────────────────────────────────

6. Monitoring Prediction Distributions

Beyond input features, monitoring the model's output distribution over time can reveal problems even without ground-truth labels available yet.

import numpy as np
 
# The model's predicted probability of the positive class, over time
predictions_week1 = np.random.beta(2, 5, 1000)      # e.g. mostly low probabilities — normal
predictions_week8 = np.random.beta(5, 2, 1000)      # e.g. now mostly HIGH probabilities — suspicious!
 
print(f"Week 1 average predicted probability: {predictions_week1.mean():.3f}")
print(f"Week 8 average predicted probability: {predictions_week8.mean():.3f}")
Why This Is a Useful Early-Warning Signal
─────────────────────────────────────────
  If a fraud model SUDDENLY starts flagging 40% of transactions
  as fraud, when it historically flagged ~2%, SOMETHING has
  changed — either genuine fraud has spiked dramatically (worth
  knowing regardless!), or (more likely) a DATA PIPELINE bug is
  feeding the model corrupted or mis-formatted input.

  This kind of PREDICTION monitoring can catch problems FASTER
  than waiting for ground-truth labels (which, per Section 4,
  can be slow or unavailable).
─────────────────────────────────────────

7. When and How to Retrain

Retraining Triggers
─────────────────────────────────────────
  Scheduled:       retrain on a FIXED CADENCE (weekly, monthly)
                     regardless of detected drift — simple,
                     predictable, but potentially wasteful or
                     insufficiently responsive

  Drift-triggered:    retrain when Section 5-6's monitoring
                        crosses a defined THRESHOLD (e.g. PSI > 0.25)
                        — more responsive, requires robust
                        monitoring infrastructure

  Performance-triggered: retrain when ACTUAL performance metrics
                            (Module 4, Ch.6), once ground truth
                            becomes available, degrade below an
                            acceptable threshold
─────────────────────────────────────────

# A conceptual retraining pipeline, tying together this ENTIRE curriculum
def retraining_pipeline(new_training_data, current_pipeline, validation_data):
    from sklearn.model_selection import cross_val_score
 
    # Retrain using Module 2's preprocessing + Modules 3-6's algorithms
    new_pipeline = current_pipeline.set_params()  # conceptually: refit on NEW data
    new_pipeline.fit(new_training_data.drop("target", axis=1), new_training_data["target"])
 
    # Evaluate the NEW model using Module 4, Ch.6 / Module 3, Ch.4's metrics
    new_scores = cross_val_score(new_pipeline, validation_data.drop("target", axis=1),
                                   validation_data["target"], cv=5)
 
    # Compare statistically against the CURRENT production model (Module 7, Ch.4)
    current_scores = cross_val_score(current_pipeline, validation_data.drop("target", axis=1),
                                        validation_data["target"], cv=5)
 
    from scipy import stats
    _, p_value = stats.ttest_rel(new_scores, current_scores)
 
    if new_scores.mean() > current_scores.mean() and p_value < 0.05:
        print("New model is SIGNIFICANTLY better — recommend A/B testing (Ch.2) before full rollout")
        return new_pipeline
    else:
        print("New model does NOT show significant improvement — keep current model")
        return current_pipeline

This single function draws on nearly the entire curriculum — Module 2's preprocessing, Modules 3-6's algorithms, Module 1/4's evaluation, Module 7's statistical model comparison, and Chapter 2's A/B testing before full rollout.

8. Building a Monitoring Dashboard

Directly reusing the Data Science Notes' visualization module — a monitoring dashboard is simply Module 4, Chapter 5's Streamlit skills applied to ongoing production health, rather than a one-time analysis.

import streamlit as st
import pandas as pd
 
st.title("Model Monitoring Dashboard")
 
col1, col2, col3 = st.columns(3)
col1.metric("Current PSI (age)", "0.08", delta="-0.02")
col2.metric("Predictions Flagged as Fraud (%)", "2.3%", delta="+0.1%")
col3.metric("Days Since Last Retrain", "12")
 
st.subheader("Prediction Distribution Over Time")
# A chart tracking Section 6's prediction distribution monitoring, week over week

9. Summary & Next Steps

Key Takeaways

  • A deployed model is never "done" — data drift (input distribution shift) and concept drift (relationship shift between inputs and target) can silently degrade performance over time.
  • Statistical tests (KS-test, PSI) detect distribution shifts systematically, reusing the Data Science Notes' statistical toolkit rather than eyeballing data manually.
  • Monitoring the model's own prediction distribution can catch problems earlier than waiting for slow or unavailable ground-truth labels.
  • Retraining can be triggered on a schedule, by detected drift, or by degraded performance — a rigorous retraining pipeline reuses this entire curriculum's evaluation and statistical comparison tools before replacing a production model.

Module 9 Complete (Concepts) — Next: Capstone

You now have the complete production ML toolkit: reproducible pipelines (Chapter 1), deployment strategies (Chapter 2), and ongoing monitoring/retraining (this chapter). Chapter 4 brings this entire curriculum together in one full, end-to-end capstone project.

Concept Check

  1. What's the difference between data drift and concept drift, and why is concept drift generally harder to detect?
  2. Why can monitoring a model's prediction distribution catch problems before ground-truth labels become available?
  3. What are the three general triggers for deciding to retrain a production model?

Next Chapter

→ Chapter 4: Capstone Project — End-to-End ML System

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index