Forward Propagation & Loss Functions

Jr Codex Deep Learning Notes

Level: Intermediate Prerequisites: Chapter 2: Activation Functions Time to complete: ~20 minutes

Table of Contents

1. Forward Propagation, Formally
2. A Full Forward Pass, Layer by Layer
3. What a Loss Function Does
4. Regression Losses
5. Classification Losses
6. Putting It Together in PyTorch
7. Summary & Next Steps

1. Forward Propagation, Formally

Forward propagation (or the "forward pass") is simply running input data through every layer of the network, in order, to produce a final output — exactly what Chapter 1's dense layers and Chapter 2's activation functions do when chained together.

Forward Propagation — Data Flowing ONE DIRECTION
─────────────────────────────────────────
  Input → Layer 1 → Activation → Layer 2 → Activation → ... → Output

  Each layer's OUTPUT becomes the NEXT layer's INPUT — data
  flows strictly forward, never backward, during this pass.
  (Backpropagation, Chapter 4, is what flows information
  BACKWARD, after the forward pass completes.)
─────────────────────────────────────────

2. A Full Forward Pass, Layer by Layer

import numpy as np
 
def relu(z):
    return np.maximum(0, z)
 
def sigmoid(z):
    return 1 / (1 + np.exp(-z))
 
# A 2-layer network: 4 inputs -> 3 hidden neurons -> 1 output
np.random.seed(42)
W1 = np.random.randn(3, 4) * 0.1
b1 = np.zeros(3)
W2 = np.random.randn(1, 3) * 0.1
b2 = np.zeros(1)
 
def forward_pass(x):
    z1 = W1 @ x + b1
    a1 = relu(z1)              # hidden layer's output, after activation
 
    z2 = W2 @ a1 + b2
    a2 = sigmoid(z2)           # output layer's output (a probability, for binary classification)
 
    return a2, (z1, a1, z2, a2)      # cache intermediate values — Chapter 4's backprop needs them
 
x = np.array([0.5, 0.3, 0.9, 0.1])
prediction, cache = forward_pass(x)
print(f"Prediction: {prediction}")

Caching every intermediate value (z1, a1, z2, a2) matters — Chapter 4's backpropagation algorithm needs these exact values to compute gradients; this is a detail that PyTorch handles automatically via its computational graph, but is essential to understand when implementing backprop by hand.

3. What a Loss Function Does

A loss function (also called a cost function or objective function) measures how wrong a single prediction is, compared to the true label — directly the same concept as the ML Notes' evaluation metrics, but computed per example, during every training step, to drive learning itself (not just to evaluate a finished model).

The Role of Loss in Training
─────────────────────────────────────────
  1. Forward pass (Section 2) produces a prediction
  2. Loss function compares prediction to TRUE label — outputs
       a single number (higher = worse)
  3. Backpropagation (Chapter 4) computes how each weight
       should change to REDUCE this loss
  4. Gradient descent (Chapter 5) actually updates the weights
  5. Repeat, over many examples and many passes through the
       data (called EPOCHS) — the loss should trend DOWNWARD
─────────────────────────────────────────

4. Regression Losses

For predicting continuous values — directly reusing the ML Notes' regression evaluation metrics.

def mean_squared_error(y_true, y_pred):
    return np.mean((y_true - y_pred) ** 2)
 
def mean_absolute_error(y_true, y_pred):
    return np.mean(np.abs(y_true - y_pred))
 
y_true = np.array([3.0, 5.0, 2.5])
y_pred = np.array([2.8, 5.2, 3.0])
 
print(f"MSE: {mean_squared_error(y_true, y_pred):.4f}")
print(f"MAE: {mean_absolute_error(y_true, y_pred):.4f}")

MSE vs MAE — the Same Tradeoff Covered in ML Notes
─────────────────────────────────────────
  MSE:  penalizes LARGE errors disproportionately (squaring) —
          sensitive to outliers, but produces smoother gradients
          for optimization
  MAE:     treats all errors proportionally — more robust to
              outliers, but its gradient is constant, which can
              slow convergence near the minimum
─────────────────────────────────────────

5. Classification Losses

Cross-entropy loss (also called log loss) is the standard loss for classification, pairing naturally with sigmoid (binary) or softmax (multi-class) output activations from Chapter 2.

def binary_cross_entropy(y_true, y_pred, epsilon=1e-15):
    y_pred = np.clip(y_pred, epsilon, 1 - epsilon)      # avoid log(0)
    return -np.mean(y_true * np.log(y_pred) + (1 - y_true) * np.log(1 - y_pred))
 
y_true = np.array([1, 0, 1])
y_pred = np.array([0.9, 0.1, 0.8])      # CONFIDENT and CORRECT predictions
print(f"BCE (good predictions): {binary_cross_entropy(y_true, y_pred):.4f}")      # LOW loss
 
y_pred_bad = np.array([0.1, 0.9, 0.2])      # CONFIDENT and WRONG predictions
print(f"BCE (bad predictions): {binary_cross_entropy(y_true, y_pred_bad):.4f}")      # HIGH loss

Why Cross-Entropy, and Not MSE, for Classification
─────────────────────────────────────────
  Cross-entropy specifically PUNISHES confident wrong
  predictions much more harshly than MSE would — a prediction
  of 0.99 for the wrong class produces a much larger cross-
  entropy loss than MSE would assign to the same error,
  producing a STRONGER, more useful gradient signal for the
  network to learn from.
─────────────────────────────────────────

def categorical_cross_entropy(y_true_onehot, y_pred_probs, epsilon=1e-15):
    y_pred_probs = np.clip(y_pred_probs, epsilon, 1)
    return -np.sum(y_true_onehot * np.log(y_pred_probs)) / len(y_true_onehot)
 
# 3-class example: true class is class 1 (index 1), one-hot encoded
y_true_onehot = np.array([0, 1, 0])
y_pred_probs = np.array([0.1, 0.7, 0.2])      # softmax output (Ch.2, Section 6)
print(f"Categorical cross-entropy: {categorical_cross_entropy(y_true_onehot, y_pred_probs):.4f}")

6. Putting It Together in PyTorch

import torch
import torch.nn as nn
 
model = nn.Sequential(
    nn.Linear(4, 3),
    nn.ReLU(),
    nn.Linear(3, 1),
)
 
x = torch.tensor([0.5, 0.3, 0.9, 0.1])
y_true = torch.tensor([1.0])
 
prediction = model(x)          # forward pass (Section 2)
 
loss_fn = nn.BCEWithLogitsLoss()      # combines sigmoid + binary cross-entropy in ONE stable operation
loss = loss_fn(prediction, y_true)
print(f"Loss: {loss.item():.4f}")
 
# Common PyTorch loss functions, by task
mse_loss = nn.MSELoss()                    # regression
bce_loss = nn.BCEWithLogitsLoss()             # binary classification
ce_loss = nn.CrossEntropyLoss()                  # multi-class classification (expects RAW logits, applies softmax internally)

BCEWithLogitsLoss and CrossEntropyLoss expect raw, un-activated logits, not post-sigmoid/softmax outputs — this is a deliberate PyTorch design choice for numerical stability (combining the activation and loss computation avoids precision issues with very confident predictions), and a common source of beginner bugs when a model's forward pass includes an unnecessary final activation.

7. Summary & Next Steps

Key Takeaways

• Forward propagation is running input data through every layer in sequence, caching intermediate values needed for backpropagation.
• A loss function quantifies how wrong a prediction is; training is the process of adjusting weights to minimize it, over many examples and epochs.
• MSE and MAE are standard regression losses; cross-entropy is the standard classification loss, chosen specifically because it penalizes confident wrong predictions more harshly than MSE would.
• PyTorch's BCEWithLogitsLoss and CrossEntropyLoss expect raw logits (not post-activation outputs) for numerical stability — a common source of bugs if misunderstood.

Concept Check

1. Why does the forward pass need to cache intermediate values like z1 and a1, rather than just the final output?
2. Why is cross-entropy loss generally preferred over MSE for classification tasks?
3. What's a common mistake when using PyTorch's CrossEntropyLoss, related to activation functions?

Next Chapter

→ Chapter 4: Backpropagation Explained

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to DL Index