Privacy & Security in AI Systems

Jr Codex AI Notes

Level: Intermediate–Advanced Prerequisites: Chapter 2 Time to complete: ~25 minutes

Table of Contents

1. Two Distinct Concerns
2. Training Data Privacy
3. Membership Inference Attacks
4. Differential Privacy
5. Adversarial Examples
6. Data Poisoning
7. Model Extraction / Stealing
8. Summary & Next Steps

1. Two Distinct Concerns

This chapter covers two related but distinct problem areas — worth separating clearly before diving in:

Privacy                                Security
─────────────────────────────         ─────────────────────────────
  Can sensitive information about        Can an attacker manipulate
  people in the TRAINING DATA be           the model's BEHAVIOR or
  extracted or inferred from the             STEAL its capabilities,
  trained model? (Sections 2-4)              even without touching the
                                              original training data?
                                              (Sections 5-7)
─────────────────────────────         ─────────────────────────────

2. Training Data Privacy

Modern models (especially deep neural networks and LLMs, covered in the Deep Learning and NLP/LLM notes) are trained on enormous datasets that may contain sensitive personal information — and there's a real risk that information leaks back out through the trained model itself.

The Core Risk
─────────────────────────────────────────
  A model trained on medical records, private messages, or
  any personal data might, under certain conditions,
  MEMORIZE and later REVEAL specific pieces of that training
  data — even though the model's INTENDED purpose was only
  to learn general PATTERNS, not to store individual records.
─────────────────────────────────────────

A Well-Documented Real Concern with Large Language Models
─────────────────────────────────────────
  Researchers have demonstrated that large language models
  can sometimes be prompted to reproduce VERBATIM snippets
  from their training data — including, in some documented
  cases, personal information that happened to appear in
  training text. This is an active area of both research and
  mitigation work in the LLM field (further context in the
  NLP/LLM Notes).
─────────────────────────────────────────

3. Membership Inference Attacks

A specific, well-studied privacy attack: given a trained model and a specific data record, can an attacker determine whether that exact record was part of the training set?

# A conceptual sketch of the membership inference intuition:
# Models often behave slightly DIFFERENTLY (typically MORE confidently)
# on data they were TRAINED on, versus data they've never seen
 
def membership_inference_signal(model_confidence_on_record):
    """
    Higher confidence MIGHT suggest the record was in the training set
    (the model has effectively "memorized" it) — this simplified signal
    is the CORE intuition behind real membership inference attacks.
    """
    high_confidence_threshold = 0.95
    return model_confidence_on_record > high_confidence_threshold

Why This Matters, Concretely
─────────────────────────────────────────
  If a model was trained on a dataset of patients with a
  SPECIFIC medical condition, and an attacker can determine
  that a SPECIFIC named individual's record was in that
  training set — this alone REVEALS that person likely has
  that condition, even without ever seeing the raw training
  data directly. The mere fact of "was this person's data
  used to train THIS model" can itself be sensitive
  information.
─────────────────────────────────────────

4. Differential Privacy

Differential privacy is a rigorous mathematical framework for training models while providing a formal, provable guarantee about how much any single individual's data could have influenced the output — directly addressing Section 3's attack.

The Core Guarantee
─────────────────────────────────────────
  A differentially private training process guarantees:
  "the model's output would look ALMOST IDENTICAL whether
   or not any ONE SPECIFIC individual's data was included
   in the training set."

  This makes membership inference (Section 3) and data
  extraction (Section 2) provably HARDER — an attacker
  cannot confidently determine much about any single
  individual, because the training process is DESIGNED to
  make one person's presence or absence barely detectable.
─────────────────────────────────────────

import random
 
def add_differential_privacy_noise(true_value, sensitivity, epsilon):
    """
    A simplified illustration of the LAPLACE MECHANISM — the
    classic way to achieve differential privacy: add carefully
    calibrated random noise to a computed value before releasing it.
 
    epsilon: the PRIVACY BUDGET — smaller epsilon = MORE noise = MORE privacy,
              but LESS accuracy (a genuine, explicit trade-off)
    """
    scale = sensitivity / epsilon
    noise = random.gauss(0, scale)      # simplified; real implementations use a Laplace distribution
    return true_value + noise
 
true_average_salary = 65000
noisy_average = add_differential_privacy_noise(true_average_salary, sensitivity=1000, epsilon=1.0)
print(f"True: {true_average_salary}, Released (noisy): {noisy_average:.0f}")

The Privacy-Accuracy Trade-off (epsilon)
─────────────────────────────────────────
  epsilon SMALL:   strong privacy guarantee, but MORE noise
                     added — the released result is LESS
                     accurate/useful

  epsilon LARGE:      weaker privacy guarantee, but LESS noise
                        — closer to the true, exact value

  Choosing epsilon is a genuine POLICY decision, not a purely
  technical one — similar in spirit to Chapter 1's fairness
  trade-offs, this requires an explicit value judgment about
  how much accuracy to sacrifice for privacy protection.
─────────────────────────────────────────

5. Adversarial Examples

A security (not privacy) concern: inputs deliberately, subtly modified to fool a model into an incorrect output — often changes imperceptible or nonsensical to a human, but which reliably break the model.

The Classic Example
─────────────────────────────────────────
  An image classifier correctly identifies a photo of a panda
  with 99% confidence.

  Add a SPECIFIC, carefully computed pattern of noise —
  imperceptible to the human eye, the image still LOOKS
  identical to a panda — and the SAME model now confidently
  (99%+) classifies it as a GIBBON.

  This isn't a random glitch — the noise is DELIBERATELY
  crafted, often using the model's OWN internal gradients
  (a technique directly related to how the Deep Learning
  Notes describe training itself) to find the smallest
  possible change that flips the prediction.
─────────────────────────────────────────

# Conceptual sketch — real adversarial example generation uses the model's
# gradients (from the Deep Learning Notes) to find a MINIMAL perturbation
def conceptual_adversarial_perturbation(image, model, target_wrong_label, epsilon=0.01):
    """
    Real techniques (like FGSM - Fast Gradient Sign Method) compute the
    gradient of the model's loss with respect to the INPUT (rather than
    the model's weights, which is what NORMAL training does), then nudge
    the input in the direction that INCREASES the loss for the correct
    label — the opposite of what training does.
    """
    pass      # actual implementation requires the Deep Learning Notes' gradient machinery

Real-World Security Implications
─────────────────────────────────────────
  Self-driving cars:      researchers have demonstrated
                             adversarial stickers on stop signs
                             that cause vision systems to
                             misclassify them
  Facial recognition:        adversarially patterned glasses
                               or makeup designed to evade or
                               spoof face-recognition systems
  Content moderation:           text/image adversarial
                                  perturbations designed to
                                  evade automated filters
                                  (connects to the NLP/LLM Notes)
─────────────────────────────────────────

6. Data Poisoning

An attack on the training process itself: an attacker deliberately injects malicious or mislabeled data into a training set, corrupting the resulting model's behavior in a targeted way.

# Conceptual illustration: a poisoning attack against a spam filter
# (directly extending Module 4, Chapter 3's Naive Bayes spam filter example)
 
poisoned_training_data = [
    (["buy", "cheap", "meds"], "spam"),          # legitimate spam example
    (["urgent", "wire", "transfer"], "ham"),       # POISONED — mislabeled as "ham"!
                                                       # to teach the model that THIS specific
                                                       # kind of scam phrasing is safe
]

Why This Matters
─────────────────────────────────────────
  Many real-world systems continuously RETRAIN on newly
  collected data (user feedback, crowd-sourced labels, web-
  scraped content) — this creates an ongoing ATTACK SURFACE
  where a malicious actor can deliberately inject bad examples
  to gradually shift the model's behavior in their favor.

  This is a DIRECT, practical consequence of Module 5, Chapter
  3's reward-hacking concern, but applied to SUPERVISED
  learning's training data rather than an RL reward function.
─────────────────────────────────────────

7. Model Extraction / Stealing

An attacker repeatedly queries a deployed model (e.g., through a public API) and uses the input-output pairs to train their own copycat model — effectively stealing the original's learned capability without ever accessing its internals or training data directly.

# Conceptual illustration
def model_extraction_attack(target_model_api, query_budget):
    """
    Repeatedly query the TARGET model with diverse inputs, collect
    its outputs, then train a NEW model on these (input, output) pairs
    — the new model can approximate the target's behavior without
    ever seeing its weights or training data.
    """
    collected_data = []
    for _ in range(query_budget):
        query_input = generate_diverse_input()
        output = target_model_api.predict(query_input)
        collected_data.append((query_input, output))
 
    stolen_model = train_new_model(collected_data)      # a "clone" of the target's behavior
    return stolen_model

Why This Is a Genuine Business/Security Concern
─────────────────────────────────────────
  Companies invest enormous resources (data collection,
  compute, engineering time — full context in the Machine
  Learning and Deep Learning Notes' production chapters)
  building proprietary models, often exposed only through a
  paid API.

  Model extraction lets a competitor approximate that
  capability at a fraction of the cost — a genuine commercial
  and security concern, motivating rate-limiting, query
  monitoring, and output watermarking as defenses.
─────────────────────────────────────────

8. Summary & Next Steps

Key Takeaways

• AI privacy concerns focus on whether sensitive training data can be extracted or inferred from a trained model (membership inference, memorization); AI security concerns focus on whether an attacker can manipulate model behavior or steal its capability.
• Differential privacy provides a formal, mathematical guarantee limiting any single individual's influence on a model's output, at an explicit accuracy cost controlled by a privacy budget (epsilon).
• Adversarial examples are deliberately crafted, often human-imperceptible input perturbations that reliably fool a model — a real concern for self-driving cars, facial recognition, and content moderation.
• Data poisoning corrupts a model by injecting malicious training examples, especially dangerous for systems that continuously retrain on new data; model extraction lets an attacker approximate a deployed model's capability purely by querying it repeatedly.

Concept Check

1. What's the difference between a privacy concern and a security concern in the context of AI systems?
2. Why does differential privacy involve an explicit trade-off, and what does the epsilon parameter control?
3. Why are continuously-retraining systems particularly vulnerable to data poisoning attacks?

Next Chapter

→ Chapter 4: AI Safety & Alignment

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index