t-SNE, UMAP & Association Rules

Jr Codex ML Notes

Level: Advanced Prerequisites: Chapter 3: Principal Component Analysis (PCA) Time to complete: ~25 minutes

PCA's Limitation: Linear Only
t-SNE — Preserving Local Neighborhoods
How t-SNE Works, Conceptually
t-SNE's Quirks and Caveats
UMAP — a Faster, More Scalable Alternative
PCA vs t-SNE vs UMAP — Choosing
Association Rule Mining — a Different Kind of Unsupervised Learning
Support, Confidence & Lift
Summary & Next Steps

1. PCA's Limitation: Linear Only

Chapter 3's PCA finds linear combinations of features — directly analogous to Module 3, Chapter 1's linear regression limitation. When the genuinely important structure in high-dimensional data is curved or non-linear, PCA can miss it entirely, echoing Module 3, Chapter 3's motivation for non-linear regression.

import numpy as np
from sklearn.decomposition import PCA
 
# Data lying on a curved "S-shape" in 3D — genuinely non-linear structure
# PCA, being fundamentally LINEAR, cannot "unroll" this curve properly —
# it can only find the best FLAT (linear) projection, which distorts
# the true underlying structure

2. t-SNE — Preserving Local Neighborhoods

t-SNE (t-Distributed Stochastic Neighbor Embedding) takes a fundamentally different approach: rather than maximizing variance (PCA), it tries to preserve which points are close neighbors to each other, even after projecting into 2D/3D.

from sklearn.manifold import TSNE
from sklearn.datasets import load_digits
import matplotlib.pyplot as plt
 
digits = load_digits()      # 64-dimensional data (8x8 pixel images of handwritten digits)
X, y = digits.data, digits.target
 
tsne = TSNE(n_components=2, random_state=42, perplexity=30)
X_embedded = tsne.fit_transform(X)
 
plt.scatter(X_embedded[:, 0], X_embedded[:, 1], c=y, cmap="tab10")
plt.title("t-SNE Visualization of Handwritten Digits")
# Often produces visually TIGHTER, more clearly SEPARATED clusters
# than PCA would for the SAME data — particularly for data with
# genuinely non-linear structure

3. How t-SNE Works, Conceptually

The Core Idea
─────────────────────────────────────────
  1. In the ORIGINAL high-dimensional space, compute how
     "similar" each pair of points is (points CLOSE together
     get HIGH similarity)

  2. Randomly place points in the LOW-dimensional (2D/3D)
     target space

  3. ITERATIVELY adjust the low-dimensional positions so that
     similar points in the ORIGINAL space end up CLOSE together
     in the NEW space too — essentially, gradient descent
     (Module 3, Ch.1) on a loss function measuring how well
     neighborhoods are preserved
─────────────────────────────────────────

PCA vs t-SNE — What Each Actually Preserves
─────────────────────────────────────────
  PCA:      preserves GLOBAL variance/structure — distances
              between FAR-APART points are meaningful, but
              local neighborhood detail can be lost (since it
              can only use LINEAR projections)

  t-SNE:       preserves LOCAL neighborhoods — points close in
                 the original space stay close in 2D, but
                 GLOBAL distances (how far apart different
                 CLUSTERS are from each other) are NOT reliably
                 preserved
─────────────────────────────────────────

4. t-SNE's Quirks and Caveats

Important Caveats — t-SNE Is Easy to Misread
─────────────────────────────────────────
  - CLUSTER SIZES in a t-SNE plot are NOT meaningful — a
    visually large cluster doesn't necessarily mean more
    actual data variance or spread
  - DISTANCES BETWEEN clusters are NOT reliably meaningful —
    two clusters plotted far apart aren't necessarily more
    "different" than two plotted closer together
  - RESULTS can vary noticeably between RUNS (due to random
    initialization) and are SENSITIVE to the `perplexity`
    hyperparameter (roughly: how many neighbors to consider) —
    always try a few perplexity values, don't trust a single run
  - COMPUTATIONALLY EXPENSIVE — doesn't scale well to very
    large datasets (tens of thousands of points can already be slow)
─────────────────────────────────────────

t-SNE is a visualization tool, not a general-purpose feature reduction technique for feeding into other algorithms — unlike PCA, its output shouldn't typically be used as input features for a downstream classifier or regressor, since its distances aren't consistently meaningful in the ways those algorithms would assume.

5. UMAP — a Faster, More Scalable Alternative

UMAP (Uniform Manifold Approximation and Projection) achieves visually similar goals to t-SNE, but with better speed, more preserved global structure, and the ability to transform new data after fitting (which t-SNE cannot do).

import umap
from sklearn.datasets import load_digits
import matplotlib.pyplot as plt
 
digits = load_digits()
X, y = digits.data, digits.target
 
reducer = umap.UMAP(n_components=2, random_state=42)
X_embedded = reducer.fit_transform(X)
 
plt.scatter(X_embedded[:, 0], X_embedded[:, 1], c=y, cmap="tab10")
plt.title("UMAP Visualization of Handwritten Digits")

UMAP's Practical Advantages Over t-SNE
─────────────────────────────────────────
  - Generally FASTER, especially on large datasets
  - Better preserves SOME global structure alongside local
    neighborhoods (a genuine improvement over t-SNE's purely
    local focus)
  - Supports `.transform()` on NEW data after fitting — t-SNE
    must be RE-RUN from scratch on any new data, since it has
    no persistent "model" the way UMAP does
─────────────────────────────────────────

In modern practice, UMAP has become the more common default choice over t-SNE for most applications — though both remain widely used, and results are often visually comparable for a given dataset.

6. PCA vs t-SNE vs UMAP — Choosing

Decision Guide
─────────────────────────────────────────
  Need FAST, INTERPRETABLE reduction, will use the output as
  INPUT to another algorithm, or want to preserve GLOBAL
  variance structure
      → PCA (Chapter 3)

  Purely for VISUALIZATION, want to reveal LOCAL cluster
  structure as clearly as possible, dataset is SMALL-MEDIUM
      → t-SNE or UMAP

  Need to visualize AND later transform NEW data with the
  SAME fitted reduction, or need BETTER SPEED on large data
      → UMAP specifically
─────────────────────────────────────────

7. Association Rule Mining — a Different Kind of Unsupervised Learning

A final, distinct unsupervised technique — rather than grouping data points (clustering) or reducing dimensions, association rule mining discovers relationships between items that frequently occur together.

The Classic Application: Market Basket Analysis
─────────────────────────────────────────
  "Customers who buy bread and butter ALSO tend to buy milk"

  Discovered PURELY from transaction data, with NO pre-existing
  labels — an unsupervised pattern-finding task, just like
  clustering, but over ITEM CO-OCCURRENCE rather than data
  point similarity.
─────────────────────────────────────────

import pandas as pd
from mlxtend.frequent_patterns import apriori, association_rules
 
transactions = pd.DataFrame({
    "bread":  [1, 1, 0, 1, 1],
    "butter": [1, 1, 1, 0, 1],
    "milk":   [1, 0, 1, 1, 1],
    "eggs":   [0, 1, 0, 0, 1],
}, dtype=bool)
 
frequent_itemsets = apriori(transactions, min_support=0.4, use_colnames=True)
print(frequent_itemsets)
 
rules = association_rules(frequent_itemsets, metric="confidence", min_threshold=0.7)
print(rules[["antecedents", "consequents", "support", "confidence", "lift"]])

8. Support, Confidence & Lift

The Three Key Metrics
─────────────────────────────────────────
  Support:       how FREQUENTLY does this itemset appear across
                   ALL transactions?
                   Support(bread, butter) = P(bread AND butter)

  Confidence:       given the ANTECEDENT occurred, how often did
                       the CONSEQUENT ALSO occur?
                       Confidence(bread → butter) = P(butter | bread)
                       (directly Chapter 1 of the AI Notes'
                       conditional probability, applied here)

  Lift:                how much MORE likely is the consequent,
                          GIVEN the antecedent, compared to its
                          BASELINE frequency alone?
                          Lift(bread → butter) = Confidence / P(butter)
                          Lift > 1: POSITIVE association
                          Lift = 1: NO association (independent)
                          Lift < 1: NEGATIVE association
─────────────────────────────────────────

# Interpreting a specific discovered rule
# "bread → butter": support=0.6, confidence=0.75, lift=1.25
#
# support=0.6:      60% of ALL transactions include BOTH bread and butter
# confidence=0.75:     75% of transactions with bread ALSO include butter
# lift=1.25:              buying bread makes buying butter 25% MORE
#                            likely than butter's baseline frequency alone

Lift is the most important metric for distinguishing genuine association from coincidence — a high-confidence rule can still be misleading if the consequent is simply a very common item overall (e.g., "buying anything → buying milk" might have high confidence purely because milk is popular, not because of any real relationship) — lift corrects for this baseline popularity.

9. Summary & Next Steps

Key Takeaways

t-SNE and UMAP address PCA's linear-only limitation by preserving local neighborhood structure through non-linear projection — powerful for visualization, but their cluster sizes and inter-cluster distances aren't reliably meaningful.
UMAP is generally faster than t-SNE, better preserves some global structure, and can transform new data after fitting — advantages t-SNE lacks.
Neither t-SNE nor UMAP output should typically be fed into downstream supervised models the way PCA's can — they're primarily visualization tools.
Association rule mining discovers which items frequently co-occur, using support (overall frequency), confidence (conditional probability), and lift (association strength relative to baseline) to identify genuine, non-coincidental relationships.

Module 6 Complete — Next Module

You now have the complete unsupervised learning toolkit: K-Means, hierarchical clustering, and DBSCAN for grouping data, plus PCA, t-SNE, UMAP, and association rules for reducing dimensions and finding item relationships. Module 7 shifts to a cross-cutting concern that applies to every supervised algorithm covered so far — systematically selecting and tuning models rather than guessing.

Concept Check

Why shouldn't cluster sizes or inter-cluster distances in a t-SNE plot be interpreted literally?
What practical advantage does UMAP have over t-SNE when new data arrives after the initial visualization?
Why is lift a more reliable indicator of a genuine association than confidence alone?

Next Chapter

→ Module 7: Model Selection & Hyperparameter Tuning

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index