Deep Learning

Dl Engineering And Capstone

PyTorch End-to-End Workflow

Every prior module used simplified, self-contained examples. This closing module addresses the engineering practices around those concepts — the difference betw

JrCodex·6 min read

Jr Codex Deep Learning Notes

Level: Advanced Prerequisites: Module 8: Generative Deep Learning Time to complete: ~30 minutes

Table of Contents

  1. From Isolated Concepts to a Real Project
  2. Custom Datasets
  3. DataLoaders in Depth
  4. Organizing a Real Training Script
  5. Checkpointing
  6. Reproducibility
  7. Summary & Next Steps

1. From Isolated Concepts to a Real Project

Every prior module used simplified, self-contained examples. This closing module addresses the engineering practices around those concepts — the difference between a notebook cell that runs once and a real, reusable, reproducible training project.

2. Custom Datasets

Real projects rarely use torchvision's built-in datasets (Module 4) — a custom Dataset class is the standard way to load your own data.

import torch
from torch.utils.data import Dataset
from PIL import Image
import pandas as pd
 
class CustomImageDataset(Dataset):
    def __init__(self, csv_file, image_dir, transform=None):
        self.labels_df = pd.read_csv(csv_file)      # e.g. columns: "filename", "label"
        self.image_dir = image_dir
        self.transform = transform
 
    def __len__(self):
        return len(self.labels_df)      # REQUIRED — tells PyTorch how many examples exist
 
    def __getitem__(self, idx):
        """REQUIRED — defines how to load ONE example, given its index"""
        row = self.labels_df.iloc[idx]
        image_path = f"{self.image_dir}/{row['filename']}"
        image = Image.open(image_path).convert("RGB")
 
        if self.transform:
            image = self.transform(image)      # Module 4, Ch.5's augmentation/preprocessing
 
        label = torch.tensor(row["label"], dtype=torch.long)
        return image, label
 
# Usage
from torchvision import transforms
transform = transforms.Compose([transforms.Resize((224, 224)), transforms.ToTensor()])
dataset = CustomImageDataset("labels.csv", "images/", transform=transform)
print(f"Dataset size: {len(dataset)}")
image, label = dataset[0]      # calls __getitem__ internally

Only two methods are required: __len__ and __getitem__. Everything else — batching, shuffling, parallel loading — is handled by wrapping this dataset in a DataLoader (Section 3).

3. DataLoaders in Depth

from torch.utils.data import DataLoader, random_split
 
# Splitting into train/validation — directly reusing the ML Notes' train/test discipline
train_size = int(0.8 * len(dataset))
val_size = len(dataset) - train_size
train_dataset, val_dataset = random_split(dataset, [train_size, val_size])
 
train_loader = DataLoader(
    train_dataset,
    batch_size=32,
    shuffle=True,           # Module 2, Ch.5 — shuffle TRAINING data every epoch
    num_workers=4,             # load data using 4 PARALLEL background processes — overlaps
                                   # data loading with GPU computation, avoiding a bottleneck
    pin_memory=True,               # speeds up CPU->GPU data transfer (Module 1, Ch.4)
)
 
val_loader = DataLoader(
    val_dataset,
    batch_size=32,
    shuffle=False,      # NEVER shuffle validation/test data (Module 4, Ch.3)
    num_workers=4,
)
Why num_workers Matters
─────────────────────────────────────────
  Without parallel workers, the GPU sits IDLE while the CPU
  loads and preprocesses the NEXT batch — a significant,
  often-overlooked bottleneck. Setting num_workers > 0
  loads future batches in PARALLEL, in the background, while
  the GPU is busy computing on the CURRENT batch — directly
  improving training THROUGHPUT without changing the model
  or math at all.
─────────────────────────────────────────

4. Organizing a Real Training Script

import torch
import torch.nn as nn
 
def train_one_epoch(model, loader, optimizer, criterion, device):
    model.train()      # Module 3, Ch.3-4
    total_loss = 0
 
    for images, labels in loader:
        images, labels = images.to(device), labels.to(device)
 
        optimizer.zero_grad()
        outputs = model(images)
        loss = criterion(outputs, labels)
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)      # Module 3, Ch.6
        optimizer.step()
 
        total_loss += loss.item()
 
    return total_loss / len(loader)
 
def validate(model, loader, criterion, device):
    model.eval()      # Module 3, Ch.3-4
    total_loss, correct, total = 0, 0, 0
 
    with torch.no_grad():
        for images, labels in loader:
            images, labels = images.to(device), labels.to(device)
            outputs = model(images)
            loss = criterion(outputs, labels)
 
            total_loss += loss.item()
            _, predicted = torch.max(outputs, 1)
            correct += (predicted == labels).sum().item()
            total += labels.size(0)
 
    return total_loss / len(loader), correct / total
 
def train(model, train_loader, val_loader, num_epochs, device):
    optimizer = torch.optim.AdamW(model.parameters(), lr=0.001, weight_decay=1e-4)
    criterion = nn.CrossEntropyLoss()
    scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=num_epochs)      # Module 3, Ch.5
 
    best_val_loss = float("inf")
 
    for epoch in range(num_epochs):
        train_loss = train_one_epoch(model, train_loader, optimizer, criterion, device)
        val_loss, val_accuracy = validate(model, val_loader, criterion, device)
        scheduler.step()
 
        print(f"Epoch {epoch+1}/{num_epochs}: train_loss={train_loss:.4f}, "
              f"val_loss={val_loss:.4f}, val_acc={val_accuracy:.4f}")
 
        if val_loss < best_val_loss:      # Module 3, Ch.4's early stopping / checkpointing concept
            best_val_loss = val_loss
            torch.save(model.state_dict(), "best_model.pt")      # Section 5

Separating train_one_epoch, validate, and train into distinct functions — rather than one large script — makes the code testable, reusable, and far easier to debug than a single monolithic training loop.

5. Checkpointing

Checkpointing saves a model's state periodically, protecting against lost progress from crashes and enabling resuming training or deploying the best version seen so far (ML Notes, Module 9's serialization discipline, applied to deep learning).

import torch
 
def save_checkpoint(model, optimizer, epoch, loss, path="checkpoint.pt"):
    torch.save({
        "epoch": epoch,
        "model_state_dict": model.state_dict(),           # the model's WEIGHTS
        "optimizer_state_dict": optimizer.state_dict(),       # the optimizer's STATE (e.g. Adam's momentum terms)
        "loss": loss,
    }, path)
 
def load_checkpoint(model, optimizer, path="checkpoint.pt"):
    checkpoint = torch.load(path)
    model.load_state_dict(checkpoint["model_state_dict"])
    optimizer.load_state_dict(checkpoint["optimizer_state_dict"])
    return checkpoint["epoch"], checkpoint["loss"]
 
# Resuming training from a checkpoint after an interruption
start_epoch, last_loss = load_checkpoint(model, optimizer, "checkpoint.pt")
print(f"Resuming from epoch {start_epoch}, last loss: {last_loss:.4f}")
Why Save the OPTIMIZER State Too, Not Just the Model
─────────────────────────────────────────
  Adam (Module 3, Ch.2) maintains RUNNING AVERAGES (momentum
  and squared-gradient terms) across training steps — if you
  resume training with a FRESH optimizer, these accumulated
  statistics are LOST, and training can behave differently
  (often worse, at least temporarily) than if training had
  never been interrupted at all.
─────────────────────────────────────────

6. Reproducibility

import torch
import numpy as np
import random
 
def set_seed(seed=42):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)      # for ALL GPUs, if using more than one
    torch.backends.cudnn.deterministic = True      # forces DETERMINISTIC (if slower) GPU operations
    torch.backends.cudnn.benchmark = False
 
set_seed(42)      # call this ONCE, at the very START of a script
Why Reproducibility Matters Practically
─────────────────────────────────────────
  Without fixed seeds, weight initialization (Module 3, Ch.1),
  data shuffling (Section 3), and dropout (Module 3, Ch.4) all
  introduce RANDOMNESS — re-running the EXACT same code can
  produce DIFFERENT results, making it genuinely difficult to
  tell whether a change you made actually IMPROVED the model,
  or whether the difference is just random variation between
  runs. Fixing seeds directly enables valid experiment
  comparison, echoing the ML Notes' statistical rigor around
  comparing models (Module 7).
─────────────────────────────────────────

7. Summary & Next Steps

Key Takeaways

  • A custom Dataset needs only __len__ and __getitem__; a DataLoader handles batching, shuffling, and parallel loading around it.
  • num_workers > 0 loads future batches in parallel with GPU computation, directly improving training throughput.
  • Separating training code into distinct functions (per-epoch training, validation, orchestration) produces testable, maintainable code rather than a monolithic script.
  • Checkpointing must save both model and optimizer state to properly resume training; fixed random seeds are essential for valid, reproducible experiment comparison.

Concept Check

  1. What are the two required methods for a custom PyTorch Dataset, and what does each do?
  2. Why does increasing num_workers in a DataLoader improve training speed without changing the model itself?
  3. Why is it insufficient to save only the model's weights when checkpointing, without also saving the optimizer's state?

Next Chapter

→ Chapter 2: GPUs, Mixed Precision & Scaling Training

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to DL Index