Lesson 2 of 3 · 16 min
Weights, Learning Rate & Gain
How a network actually improves: measure the error, then take a step. Meet the step size (learning rate), the stretch knob (gain), and the scoreboard (loss).
Learning is guess, check, nudge
A brand-new network guesses randomly. Training is a loop:
1. Guess an answer.
2. Check how wrong it was.
3. Nudge every weight a little in the direction that makes it less wrong.
Repeat a few thousand times and the guesses get good. That is all 'learning' means here.
Loss: the scoreboard for wrongness
We need a number for 'how wrong'. That number is the loss (also called error).
The simplest version squares the difference between the guess and the correct answer.
Squaring makes the sign disappear, so guessing 0.20 too high costs the same as 0.20 too low — and being very wrong costs a lot.
Loss = (guess − correct)²
Guess 0.80 when the correct answer is 0.50? Loss = (0.80 − 0.50)² = 0.09.
The learning rate: how big is your step?
To fix the loss, the network checks which way the weight should move, then takes a step.
The learning rate (often written α, 'alpha') is the step size.
Worked example — walking toward the answer 0.50: start at 0.80. The error is 0.30.
With a learning rate of 0.5 the new guess is 0.80 − 0.5 × 0.30 = 0.65.
Now the error is 0.15, so the next guess is 0.65 − 0.5 × 0.15 = 0.58.
Then 0.58 − 0.5 × 0.08 = 0.54, then 0.52 — each step is smaller, and we settle onto 0.50.
w ← w − α · (slope of the loss)
Move each weight a little against the slope. α is the learning rate; the slope says which way is downhill.
Gain: the stretch before the bend
Gain multiplies the input *before* the activation function sees it.
It controls how steeply a neuron reacts.
With gain = 1, ReLU of 0.4 is 0.4. With gain = 2, the neuron sees 0.8 first, so it fires twice as hard for the same input.
Gain and weights both scale signals — the difference is that gain sits right at the activation's doorway.
output = activation(gain × sum)
Gain stretches the sum; the activation decides how much of that stretch becomes output.
Try it: change the gain
Activation Curve Playground
See how ReLU, Sigmoid and Tanh bend the same input. Slide the input dot along the curve and watch the output value change.
A neuron first multiplies its input by a weight and adds a bias. The activation function is the last step: it decides the shape of the 'boom'. Gain stretches the input before the curve sees it.
Control knobs
Check yourself
Learning Rate, Gain & Loss
What does the *learning rate* (α) control?
Put it to work: steady the beam
Balance the Ball on the Beam
The beam pushes the ball away from the middle and a slow drift makes it wander. Tune the controller weights (and the learning-rate gain) until the ball stays in the green zone — without correct tuning it slides off within a few seconds.
🎯 Goal: Change at least one knob, then hold the ball inside the green zone for 3 seconds.
Hyperparameters