Convolutional Neural Networks
Data Augmentation for Vision
Module 3, Chapter 4 introduced data augmentation as a general regularization concept. Vision is where it's most powerfully and naturally applicable — many trans
Jr Codex Deep Learning Notes
Level: Intermediate Prerequisites: Chapter 4: Object Detection & Segmentation Time to complete: ~15 minutes
Table of Contents
- Why Augmentation Matters Especially for Vision
- Common Augmentation Techniques
- Augmentation Must Preserve Labels
- Implementing Augmentation in PyTorch
- Test-Time Augmentation
- Advanced Augmentation Strategies
- Summary & Next Steps
1. Why Augmentation Matters Especially for Vision
Module 3, Chapter 4 introduced data augmentation as a general regularization concept. Vision is where it's most powerfully and naturally applicable — many transformations of an image (rotating it slightly, flipping it, adjusting brightness) don't change what is depicted, only how it's presented, giving the network many "free," genuinely different-looking training examples from each original image.
Augmentation Directly Targets This Module's Central Problem
─────────────────────────────────────────
Chapter 1, Section 1 noted CNNs need translation invariance
— but a fixed convolutional architecture only provides
invariance to SMALL positional shifts. Augmentation
extends this further, teaching the network invariance to
ROTATION, SCALE, LIGHTING, and other variations that occur
naturally in real-world images but might not be well
represented in a limited training set.
─────────────────────────────────────────
2. Common Augmentation Techniques
Standard Augmentations for Natural Images
─────────────────────────────────────────
Horizontal flip: mirrors the image left-right — valid
for most natural images (a flipped
cat is still clearly a cat)
Random crop: crops a random sub-region, then
resizes back to the target size
— teaches robustness to
framing/scale variation
Rotation: rotates by a small random angle
(e.g. ±15°) — teaches
robustness to camera angle
Color jitter: randomly adjusts brightness,
contrast, saturation —
teaches robustness to
lighting conditions
Random erasing/ randomly masks out a
cutout: small rectangular
region — a technique
conceptually similar
to dropout (Module
3, Ch.4), applied at
the INPUT level
instead of inside
the network
─────────────────────────────────────────
3. Augmentation Must Preserve Labels
A critical, domain-specific constraint: not every transformation is safe for every task.
Augmentation Choices Depend ENTIRELY on the Task
─────────────────────────────────────────
Digit recognition (e.g. MNIST): a VERTICAL flip of "6"
produces something that
looks like "9" — the
label is no longer
correct! Vertical flips
would be a HARMFUL
augmentation here
Medical imaging: rotation might be fine,
but COLOR adjustments
could remove
diagnostically
meaningful information
— augmentation choices
need DOMAIN expertise,
not just generic
defaults
Object detection (Ch.4): any spatial
augmentation
(crop, flip,
rotate) must
ALSO transform
the bounding
box coordinates
correspondingly
— a common
source of subtle
bugs
─────────────────────────────────────────
4. Implementing Augmentation in PyTorch
import torch
from torchvision import transforms
train_transform = transforms.Compose([
transforms.RandomHorizontalFlip(p=0.5),
transforms.RandomRotation(degrees=15),
transforms.RandomResizedCrop(size=224, scale=(0.8, 1.0)),
transforms.ColorJitter(brightness=0.2, contrast=0.2, saturation=0.2),
transforms.ToTensor(),
transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]), # ImageNet's known statistics
])
# CRITICALLY — the TEST/VALIDATION transform does NOT include random augmentations,
# only the necessary resizing/normalization — echoing Chapter 3's train/test distinction
test_transform = transforms.Compose([
transforms.Resize(256),
transforms.CenterCrop(224),
transforms.ToTensor(),
transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])Applying random augmentation to the test/validation set would be a mistake — evaluation needs to be deterministic and reproducible, reflecting the same preprocessing the model will see in production (ML Notes, Module 9), not artificially randomized.
5. Test-Time Augmentation
A related but distinct technique: Test-Time Augmentation (TTA) deliberately applies multiple augmented versions of a test image, gets a prediction for each, and averages them — trading extra inference compute for a small, often reliable accuracy boost.
def predict_with_tta(model, image, transform_list, device):
predictions = []
for transform in transform_list:
augmented = transform(image).unsqueeze(0).to(device)
with torch.no_grad():
output = model(augmented)
predictions.append(torch.softmax(output, dim=1))
return torch.mean(torch.stack(predictions), dim=0) # average across augmented versionsThis differs from Section 4's training-time augmentation in purpose: training augmentation prevents overfitting; TTA improves a fixed, already-trained model's inference-time robustness, at the cost of multiple forward passes per prediction — a latency tradeoff worth weighing against the ML Notes' production latency discussion.
6. Advanced Augmentation Strategies
Beyond Individual Transformations
─────────────────────────────────────────
Mixup: blends TWO training images (and their
labels) together proportionally — e.g.
70% cat image + 30% dog image, with a
correspondingly blended 70/30 label —
encourages smoother decision boundaries
CutMix: cuts a patch from one image and pastes
it onto another, blending the labels
proportionally to the patch area —
combines cutout's regional masking
with mixup's label blending
AutoAugment/ LEARNS an optimal augmentation
RandAugment: POLICY automatically (which
transforms, and how strongly,
to apply) rather than
requiring a human to
hand-pick them — an
application of automated
hyperparameter search (ML
Notes, Module 7) to the
augmentation pipeline itself
─────────────────────────────────────────
7. Summary & Next Steps
Key Takeaways
- Data augmentation is especially powerful for vision because many geometric and photometric transformations preserve an image's label while creating genuinely different-looking training examples.
- Augmentation choices must be domain- and task-appropriate — a transformation that's safe for one problem (e.g. horizontal flips) can be actively harmful for another (e.g. digit recognition).
- Training augmentation should never be applied to validation/test data, which must remain deterministic and representative of production preprocessing.
- Test-Time Augmentation averages predictions across multiple augmented versions of a test image, trading extra inference compute for a modest accuracy improvement.
Module 4 Complete — What's Next
You've now covered the complete computer vision pipeline: the convolution and pooling operations underlying CNNs, the architectural evolution from LeNet to ResNet, a full practical classification workflow, object detection and segmentation, and data augmentation. Module 5 shifts focus to sequential data — text, time series, and audio — where a different architectural family (RNNs and LSTMs) is needed.
Concept Check
- Why would a vertical flip be a poor augmentation choice for a handwritten digit classifier?
- Why should validation and test data never receive random augmentation, unlike training data?
- What's the key difference in purpose between training-time augmentation and test-time augmentation (TTA)?
Next Module
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to DL Index