Convolutional Neural Networks
Object Detection & Segmentation
Chapter 3's classifier answers "what is the single main object in this image?" Many real applications need more: "what objects are present, and where is each on
Jr Codex Deep Learning Notes
Level: Intermediate–Advanced Prerequisites: Chapter 3: Image Classification in Practice Time to complete: ~25 minutes
Table of Contents
- Beyond Whole-Image Classification
- Object Detection — Localizing AND Classifying
- Two-Stage Detectors: R-CNN Family
- One-Stage Detectors: YOLO and SSD
- Evaluating Detectors: IoU and mAP
- Semantic vs Instance Segmentation
- A Practical Detection Example
- Summary & Next Steps
1. Beyond Whole-Image Classification
Chapter 3's classifier answers "what is the single main object in this image?" Many real applications need more: "what objects are present, and where is each one?" (object detection), or "which exact pixels belong to which object?" (segmentation). This chapter is a practical orientation to both — the field is large enough to warrant dedicated study beyond this curriculum, but understanding the core ideas and terminology is essential for any computer vision practitioner.
2. Object Detection — Localizing AND Classifying
Object detection requires predicting, for each object in an image: a bounding box (the object's location and size) and a class label (what it is).
Object Detection Output, Conceptually
─────────────────────────────────────────
Input: one image, potentially containing MULTIPLE
objects
Output: a LIST of detections, each with:
- bounding box: (x, y, width, height)
- class label: e.g. "dog," "car"
- confidence score: how certain the model is
Example: [
{"box": (50, 30, 100, 150), "class": "dog", "confidence": 0.94},
{"box": (200, 60, 80, 90), "class": "car", "confidence": 0.87},
]
─────────────────────────────────────────
3. Two-Stage Detectors: R-CNN Family
The Two-Stage Approach
─────────────────────────────────────────
Stage 1: propose a set of CANDIDATE regions that might
contain an object ("region proposals")
Stage 2: for EACH candidate region, run Chapter 3's
classification approach to determine WHAT
object it contains (or "background," if
none)
─────────────────────────────────────────
The R-CNN Family's Evolution
─────────────────────────────────────────
R-CNN (2014): used a slow, separate algorithm to
propose regions, then ran a full CNN
on EACH region independently — very
slow (minutes per image)
Fast R-CNN: ran the CNN ONCE on the whole image,
then extracted region FEATURES
from the shared result — much
faster
Faster R-CNN: replaced the slow, separate
region-proposal algorithm with
a small NEURAL NETWORK
(a "Region Proposal Network")
— end-to-end trainable,
significantly faster still
─────────────────────────────────────────
Two-stage detectors tend to be more accurate but slower than one-stage alternatives (Section 4) — a direct accuracy/speed tradeoff, echoing the ML Notes, Module 9's latency discussion about algorithm choice depending on production requirements.
4. One-Stage Detectors: YOLO and SSD
YOLO ("You Only Look Once") and SSD (Single Shot Detector) skip the separate region-proposal stage entirely, predicting bounding boxes and classes directly from the image in a single pass — dramatically faster, at some accuracy cost, making them well suited to real-time applications (autonomous driving, live video analysis).
YOLO's Core Idea (Conceptual)
─────────────────────────────────────────
Divide the image into a GRID. For EACH grid cell,
simultaneously predict:
- whether an object's CENTER falls in this cell
- the bounding box coordinates, if so
- the class probabilities, if so
This turns detection into a SINGLE forward pass through one
network — no separate proposal stage, no per-region
re-processing — which is exactly why it's fast enough for
real-time use.
─────────────────────────────────────────
5. Evaluating Detectors: IoU and mAP
Intersection over Union (IoU) measures how well a predicted bounding box overlaps with the true (ground truth) box.
def calculate_iou(box1, box2):
"""boxes as (x1, y1, x2, y2) — top-left and bottom-right corners"""
x1_inter = max(box1[0], box2[0])
y1_inter = max(box1[1], box2[1])
x2_inter = min(box1[2], box2[2])
y2_inter = min(box1[3], box2[3])
intersection = max(0, x2_inter - x1_inter) * max(0, y2_inter - y1_inter)
area1 = (box1[2] - box1[0]) * (box1[3] - box1[1])
area2 = (box2[2] - box2[0]) * (box2[3] - box2[1])
union = area1 + area2 - intersection
return intersection / union if union > 0 else 0
predicted_box = (50, 30, 150, 180)
true_box = (55, 35, 145, 175)
print(f"IoU: {calculate_iou(predicted_box, true_box):.3f}") # e.g. 0.85 — a GOOD matchA detection is typically counted as "correct" if IoU exceeds a threshold (commonly 0.5). Mean Average Precision (mAP) then aggregates precision/recall (directly reusing the ML Notes' classification evaluation concepts) across all classes and IoU thresholds into a single summary metric, the standard way detector performance is reported and compared.
6. Semantic vs Instance Segmentation
Three Levels of Visual Understanding
─────────────────────────────────────────
Classification (Chapter 3): "this image contains a dog"
— one label, whole image
Object Detection (Sections 2-4): "there's a dog at THIS
bounding box" — location
+ label, per object
Semantic Segmentation: "THESE EXACT PIXELS
belong to a dog" —
pixel-level labels,
but does NOT
distinguish between
TWO separate dogs
(both just labeled
"dog")
Instance Segmentation: pixel-level labels
that ALSO
distinguish
individual
objects — "dog #1"
vs "dog #2" get
different masks
─────────────────────────────────────────
U-Net is the standard architecture for semantic segmentation — it uses an encoder (downsampling, like Chapter 1-2's CNNs) followed by a decoder (upsampling back to full resolution), with skip connections between matching encoder/decoder levels, conceptually related to ResNet's residual connections (Chapter 2). Mask R-CNN extends Faster R-CNN (Section 3) to also predict a segmentation mask for each detected object, providing instance segmentation.
7. A Practical Detection Example
import torch
from torchvision.models.detection import fasterrcnn_resnet50_fpn
from torchvision.transforms import functional as F
from PIL import Image
# Using a PRETRAINED detector directly — Module 7 covers transfer learning in depth
model = fasterrcnn_resnet50_fpn(pretrained=True)
model.eval()
image = Image.open("street_scene.jpg")
image_tensor = F.to_tensor(image).unsqueeze(0)
with torch.no_grad():
predictions = model(image_tensor)
boxes = predictions[0]["boxes"]
labels = predictions[0]["labels"]
scores = predictions[0]["scores"]
# Keep only CONFIDENT detections
confidence_threshold = 0.7
for box, label, score in zip(boxes, labels, scores):
if score > confidence_threshold:
print(f"Detected class {label.item()} at {box.tolist()} with confidence {score.item():.2f}")8. Summary & Next Steps
Key Takeaways
- Object detection requires predicting both a bounding box and a class label for every object in an image, unlike Chapter 3's whole-image classification.
- Two-stage detectors (R-CNN family) propose regions then classify them, favoring accuracy; one-stage detectors (YOLO, SSD) predict everything in a single pass, favoring speed.
- IoU measures bounding box overlap quality; mAP aggregates precision/recall across classes into the standard detector evaluation metric.
- Segmentation extends detection to pixel-level precision — semantic segmentation labels every pixel by class, while instance segmentation additionally distinguishes individual object instances.
Concept Check
- What is the fundamental tradeoff between two-stage and one-stage object detectors?
- How does IoU determine whether a detection counts as "correct"?
- What's the key difference between semantic segmentation and instance segmentation?
Next Chapter
→ Chapter 5: Data Augmentation for Vision
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to DL Index