Backpropagation & Gradient Descent in Depth

Before a network can learn, it must make a guess. That guess-making trip is the forward pass: numbers enter on the left and flow right.

A tiny 2-2-1 network. Inputs x₁ = 1.0, x₂ = 0.5.

Hidden neuron 1: 0.5 × 1.0 + 0.8 × 0.5 + 0.1 = 0.5 + 0.4 + 0.1 = 1.0. ReLU keeps it: h₁ = 1.0.

Hidden neuron 2: −0.4 × 1.0 + 0.2 × 0.5 + 0.2 = −0.4 + 0.1 + 0.2 = −0.1. ReLU flattens it: h₂ = 0.

Output: 0.7 × 1.0 + (−0.5) × 0 + 0.0 = 0.7. Sigmoid turns that into 1 / (1 + e^(−0.7)) = 0.668.

So the network guesses about 0.67.

A loss function turns the gap between guess and target into a number. The classic is mean squared error (MSE).

Worked MSE with 2 data points:

Point 1: guess 0.8, target 1.0 → (0.8 − 1.0)² = 0.04.

Point 2: guess 0.4, target 0.5 → (0.4 − 0.5)² = 0.01.

MSE = (0.04 + 0.01) / 2 = 0.025. Smaller is better; zero would be perfect.

To improve a weight, we need its slope: if I nudge this knob up, does the loss go up or down, and how fast?

No calculus required — we can just measure it. Take the loss L(w) = (w − 1)² (the guess is w, the target is 1).

At w = 0.5: L = (0.5 − 1)² = 0.25.

At w = 0.51: L = (0.51 − 1)² = (−0.49)² = 0.2401.

Slope ≈ (0.2401 − 0.25) / (0.51 − 0.5) = −0.0099 / 0.01 = −0.99.

The slope is about −1.0. Negative means nudging w up makes the loss go down — so we should nudge up.

The exact calculus answer is 2(w − 1) = 2(0.5 − 1) = −1.0, matching our finite-difference estimate.

The update rule says: move each weight a little against the slope.

w_new = w_old − α × slope, where α is the learning rate.

Let us run it on L(w) = (w − 1)², whose slope is 2(w − 1). Start at w = 3.0, use α = 0.2.

Iteration 1: slope 2(3.0 − 1) = 4.0; w = 3.0 − 0.2 × 4.0 = 2.20; loss (1.20)² = 1.44.

Iteration 2: slope 2(2.20 − 1) = 2.40; w = 2.20 − 0.48 = 1.72; loss (0.72)² = 0.5184.

Iteration 3: slope 2(1.72 − 1) = 1.44; w = 1.72 − 0.288 = 1.432; loss (0.432)² ≈ 0.187.

Iteration 4: slope 2(1.432 − 1) = 0.864; w = 1.432 − 0.1728 = 1.259; loss (0.259)² ≈ 0.067.

The steps shrink as the slope flattens: 3.0 → 2.2 → 1.72 → 1.43 → 1.26 → …, marching toward the minimum at w = 1.

A deep network has many weights, each with its own slope. Backpropagation computes all of them efficiently with one backward sweep.

Blame assignment analogy. When a project fails, you trace the mistake back up the chain to see who contributed. Backprop does the same: start with the output error, then ask every weight 'how much did you push the output?'

The tool is the chain rule: to find how a change travels through a chain, multiply the small effects along the way.

∂Loss/∂w = (∂Loss/∂ŷ) × (∂ŷ/∂z) × (∂z/∂w).

Target y = 1.0, guess ŷ = 0.668, and loss ½(ŷ − y)².

Output signal: ∂Loss/∂ŷ = ŷ − y = 0.668 − 1.0 = −0.332.

Through the sigmoid: ∂ŷ/∂z = ŷ(1 − ŷ) = 0.668 × 0.332 = 0.222. Multiply: δ_out = −0.332 × 0.222 = −0.0736.

Output weight `v₁`: the output was z = v₁h₁ + v₂h₂ + b₀ and h₁ = 1.0, so ∂z/∂v₁ = 1.0. Thus ∂Loss/∂v₁ = −0.0736. With α = 0.5: v₁ ← 0.7 − 0.5 × (−0.0736) = 0.7368.

Hidden weight `w₁`: blame keeps flowing back. ∂z/∂h₁ = v₁ = 0.7; ReLU's slope is 1 because h₁ > 0; and ∂z₁/∂w₁ = x₁ = 1.0.

So ∂Loss/∂w₁ = −0.0736 × 0.7 × 1 × 1.0 = −0.0515, and w₁ ← 0.5 − 0.5 × (−0.0515) = 0.5258.

One backward sweep gave every weight its own update — that is the whole trick.

Backprop multiplies slopes layer by layer. If each slope is smaller than 1, the product shrinks fast.

Sigmoid's slope is at most 0.25 (at x = 0).

Five sigmoid layers: 0.25⁵ ≈ 0.001.

Ten sigmoid layers: 0.25¹⁰ ≈ 0.000001.

By the time the blame reaches the early layers it is almost zero, so those weights barely move — the vanishing-gradient problem. That is why ReLU (slope 1 for positives) and shortcuts like skip connections made deep networks practical.

Slopes above 1, on the other hand, can compound into exploding gradients — which is one reason huge learning rates blow a deep network up.