Lesson 3 of 3 · 13 min
Activation Functions: ReLU, Sigmoid & Tanh
Three favourite firing rules, with the real formulas and a hand-worked example for each.
Why bend the signal?
If a network only added and multiplied, stacking layers would still behave like one straight line.
Activation functions add the bend that lets networks learn curvy, interesting patterns.
Feed a number in, get a number out — the activation is the shape of the doorway the number must pass through.
ReLU(x) = max(0, x)
Keep positives, flatten negatives to zero.
ReLU by hand
Plug in x = -2: max(0, -2) = 0.
Plug in x = 0: max(0, 0) = 0.
Plug in x = 2: max(0, 2) = 2.
Simple rule: if the sum is positive it passes straight through, otherwise the neuron stays silent.
Sigmoid(x) = 1 / (1 + e^(−x))
Squashes any input into the range 0 to 1.
Sigmoid by hand
Plug in x = -2: 1 / (1 + e²) = 1 / 8.39 = 0.12 — pretty sure it is *not* firing.
Plug in x = 0: 1 / (1 + 1) = 0.50 — an exact maybe.
Plug in x = 2: 1 / (1 + e⁻²) = 1 / 1.14 = 0.88 — very likely firing.
Because the output always lands between 0 and 1, sigmoid is perfect when you want a probability.
Tanh(x) = (e^x − e^(−x)) / (e^x + e^(−x))
Like sigmoid, but centred on zero: output lands between -1 and 1.
Tanh by hand
Plug in x = -2: tanh(-2) = -0.96 — strongly negative.
Plug in x = 0: tanh(0) = 0.00 — neutral.
Plug in x = 2: tanh(2) = 0.96 — strongly positive.
Because zero sits in the middle, tanh often helps networks learn faster than sigmoid.
Try it: bend the curve
Activation Curve Playground
See how ReLU, Sigmoid and Tanh bend the same input. Slide the input dot along the curve and watch the output value change.
A neuron first multiplies its input by a weight and adds a bias. The activation function is the last step: it decides the shape of the 'boom'. Gain stretches the input before the curve sees it.
Control knobs
Check yourself
Activation Functions
Which activation function outputs exactly zero for every negative input?
Bonus: steer the rover
Drive the Rover to the Flag
The rover drives in a straight line by default — and sails right past the flag. Pause, tune the steering weights, then run again. Hazard rings reset the rover to the start, and the target shifts a little every run.
🎯 Goal: Change at least one knob, then reach the flag within 320 steps without hitting a hazard ring.
Hyperparameters