Model Deployment Basics

Jr Codex ML Notes

Level: Advanced Prerequisites: Chapter 1: Building ML Pipelines Time to complete: ~25 minutes

Table of Contents

1. From a Saved File to a Live Service
2. Batch vs Real-Time Prediction
3. Serving Predictions with a REST API
4. Input Validation at the Boundary
5. Latency Considerations
6. Containerization — a Brief Orientation
7. A/B Testing a New Model Version
8. Summary & Next Steps

1. From a Saved File to a Live Service

Chapter 1 ended with a serialized pipeline file. This chapter covers making it accessible to real applications, users, or other systems that need predictions — a genuinely different skill set from model building itself, closer to software engineering than data science.

2. Batch vs Real-Time Prediction

The first deployment decision, driven entirely by how the predictions will actually be used.

Batch Prediction
─────────────────────────────────────────
  Run predictions on a LARGE set of examples PERIODICALLY
  (e.g. nightly), store the results, and serve them from
  storage when needed.

  Good for: recommendation pre-computation, periodic risk
              scoring, reports — anything where predictions
              don't need to reflect data from the last few
              seconds
─────────────────────────────────────────

Real-Time (Online) Prediction
─────────────────────────────────────────
  Serve a prediction IMMEDIATELY, in response to a SPECIFIC
  request, using the CURRENT state of the input.

  Good for: fraud detection at transaction time, live
              recommendation as a user browses — anything
              needing the FRESHEST possible input
─────────────────────────────────────────

# Batch prediction — straightforward, reusing Chapter 1's pipeline directly
import joblib
import pandas as pd
 
pipeline = joblib.load("model_pipeline.joblib")
new_batch = pd.read_csv("todays_new_customers.csv")
 
predictions = pipeline.predict(new_batch)
pd.DataFrame({"customer_id": new_batch["customer_id"], "prediction": predictions}).to_csv(
    "predictions_output.csv", index=False
)

Choosing Between Them
─────────────────────────────────────────
  Batch:      simpler infrastructure, easier to debug, fine
                when some DELAY between "new data arrives" and
                "prediction available" is acceptable

  Real-time:     more infrastructure complexity (Sections 3-5),
                    but necessary when a prediction is needed
                    THE MOMENT new data arrives
─────────────────────────────────────────

3. Serving Predictions with a REST API

The standard way to offer real-time predictions to other systems — wrapping the pipeline in a lightweight web service.

from fastapi import FastAPI
from pydantic import BaseModel
import joblib
import numpy as np
 
app = FastAPI()
pipeline = joblib.load("model_pipeline.joblib")      # loaded ONCE, at startup — not per-request
 
class PredictionRequest(BaseModel):
    age: float
    income: float
    city: str
 
@app.post("/predict")
def predict(request: PredictionRequest):
    import pandas as pd
    input_df = pd.DataFrame([request.dict()])
    prediction = pipeline.predict(input_df)[0]
    probability = pipeline.predict_proba(input_df)[0].max()
 
    return {
        "prediction": str(prediction),
        "confidence": float(probability),
    }

uvicorn app:app --host 0.0.0.0 --port 8000

# Any client — a web app, another service, a mobile app backend —
# can now get a prediction via a simple HTTP request:
import requests
response = requests.post("http://localhost:8000/predict", json={
    "age": 34, "income": 65000, "city": "Boston"
})
print(response.json())      # {"prediction": "1", "confidence": 0.87}

Loading the pipeline once at startup, not per-request, is a critical performance detail — reloading a serialized model file on every single prediction request would add unnecessary, avoidable latency (Section 5).

4. Input Validation at the Boundary

Directly connects to the AI Notes' discussion of agents perceiving an uncertain environment — a production API receives input from the outside world, which cannot be trusted to always match training-data assumptions.

from pydantic import BaseModel, field_validator
 
class PredictionRequest(BaseModel):
    age: float
    income: float
    city: str
 
    @field_validator("age")
    @classmethod
    def age_must_be_reasonable(cls, v):
        if not (0 <= v <= 120):
            raise ValueError("age must be between 0 and 120")
        return v
 
    @field_validator("income")
    @classmethod
    def income_must_be_non_negative(cls, v):
        if v < 0:
            raise ValueError("income cannot be negative")
        return v

Why This Matters — Connecting to Earlier Modules
─────────────────────────────────────────
  Module 2, Ch.2's OneHotEncoder(handle_unknown="ignore")
  already prepared for an UNSEEN category at prediction time —
  but validation at the API boundary catches OTHER problems
  BEFORE they even reach the pipeline: negative ages, malformed
  types, missing required fields — directly echoing the Data
  Science Notes' data cleaning chapter's validation discussion,
  now applied at a LIVE, real-time boundary instead of a batch
  file.
─────────────────────────────────────────

Never trust that production input will look like training data — validating and rejecting malformed requests explicitly, with a clear error message, is far better than letting bad input silently produce a nonsensical or crashing prediction.

5. Latency Considerations

For real-time serving specifically, how fast a prediction can be returned is often as important as the model's accuracy — directly connecting to Module 4, Chapter 2's KNN prediction-cost discussion and Module 5, Chapter 3's XGBoost speed advantages.

Latency Considerations by Algorithm (Recalling Earlier Modules)
─────────────────────────────────────────
  KNN (Module 4, Ch.2):        SLOW at prediction time — must
                                  compare against the entire
                                  stored training set for EVERY
                                  request — often a poor fit for
                                  low-latency real-time serving

  Linear/Logistic Regression       FAST — a single, simple
  (Module 3-4):                     computation per prediction

  Decision Trees / Random             FAST — a sequence of simple
  Forests (Module 4-5):                 threshold comparisons

  Deep Neural Networks                    can be SLOWER, but
  (Deep Learning Notes):                    modern optimization
                                              techniques (covered
                                              there) address this
─────────────────────────────────────────

This is a genuine, practical reason algorithm choice (Modules 3-6) sometimes trades off against accuracy — the single most accurate model in offline evaluation isn't automatically the right production choice if its prediction latency is unacceptable for the actual use case.

6. Containerization — a Brief Orientation

Worth knowing exists, even without full coverage here: Docker packages an application (the API from Section 3, its dependencies, and the model file) into a portable, consistent unit that runs identically across different environments.

# A conceptual Dockerfile sketch
FROM python:3.11-slim
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY model_pipeline.joblib .
COPY app.py .
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

Why Containerization Matters for ML Specifically
─────────────────────────────────────────
  A model trained with scikit-learn version X, on Python
  version Y, with specific NumPy behavior, might behave
  SUBTLY differently on a different environment — containerization
  freezes the ENTIRE environment (not just the model file),
  guaranteeing the model behaves consistently between where it
  was developed and where it's deployed.
─────────────────────────────────────────

7. A/B Testing a New Model Version

Directly reusing the Data Science Notes' A/B testing chapter — before fully replacing a production model with a new, "improved" version, test it properly.

Why This Matters, Even After Rigorous Offline Evaluation (Module 7)
─────────────────────────────────────────
  Module 7's cross-validation and hyperparameter tuning
  evaluate a model on HISTORICAL data — but production
  behavior can differ subtly (Chapter 3's data drift), and
  business metrics (actual revenue, user satisfaction) don't
  always move in lockstep with offline metrics (Module 4, Ch.6's
  F1, accuracy, etc.).

  The Data Science Notes' A/B testing chapter's ENTIRE
  toolkit — randomized assignment, pre-registered primary
  metrics, sample size calculation, checking for Sample Ratio
  Mismatch — applies DIRECTLY to comparing "old model" vs "new
  model" in a live production setting.
─────────────────────────────────────────

import random
 
def route_prediction_request(user_id, old_pipeline, new_pipeline, new_model_traffic_pct=0.1):
    """Route a small percentage of REAL traffic to the new model, rest to the old."""
    if hash(user_id) % 100 < new_model_traffic_pct * 100:
        return new_pipeline.predict, "new_model"
    return old_pipeline.predict, "old_model"

8. Summary & Next Steps

Key Takeaways

• Batch prediction suits periodic, non-urgent needs; real-time prediction serves individual requests immediately, requiring more infrastructure (a REST API, careful latency management).
• A production API should load the model pipeline once at startup, not per-request, and should validate incoming requests explicitly rather than trusting they'll match training-data assumptions.
• Prediction latency is a genuine deployment concern that can influence algorithm choice — KNN's slow per-prediction cost (Module 4, Chapter 2) is a real production consideration, not just a theoretical one.
• Containerization (Docker) freezes an entire runtime environment, ensuring a model behaves consistently between development and production.
• Before fully replacing a production model, A/B test it using the Data Science Notes' experimentation toolkit — offline evaluation metrics don't always predict real-world business impact.

Concept Check

1. What's the key difference in requirements between batch and real-time prediction, and how does that difference influence infrastructure choices?
2. Why should input validation happen at the API boundary rather than trusting the pipeline's preprocessing to handle anything?
3. Why might a model's offline accuracy not fully predict its real-world business impact, and what technique addresses this gap?

Next Chapter

→ Chapter 3: Model Monitoring, Drift & Retraining

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index