Backpropagation from scratch

A model predicts that a support ticket is safe to wait. The target says it was urgent. Knowing the prediction was wrong is not enough: the system needs to know which of thousands or billions of adjustable numbers should change, and in which direction.

BackpropagationBackpropagationAn efficient method for calculating how a model’s loss changes with respect to each parameter. It works backwards through a computation graph, combining local derivatives with the chain rule. Backpropagation calculates gradients; an optimiser uses them to update parameters. is the accounting method that assigns that local credit. It is the bridge between a forward-pass calculator and a model that can learn.

Why this matters

Every trainable neural network needs a way to improve its parameters from examples. Backpropagation makes this practical: it calculates a gradient for each parameter without separately redoing the whole calculation for every parameter. Frameworks automate the arithmetic, but debugging a training run requires knowing what they are doing.

Plain-English mental model

Step 1 — Turn an error into a number the model can reduce

A model needs a numeric target for improvement. A loss functionLoss functionA numerical score describing how costly a model's prediction is relative to the training target. Fitting adjusts parameters to reduce an aggregate training loss, but lower training loss does not guarantee better real-world behaviour. measures how costly a prediction is compared with the training target. For one numeric prediction, a simple choice is squared error:

loss = (prediction − target)²

Let input x = 2, target t = 1, and one parameter w = 0. Our tiny model predicts:

y = w × x = 0 × 2 = 0
error = y − t = 0 − 1 = −1
loss = error² = (−1)² = 1

The loss is not a moral judgement and it is not the same as user harm. It is the training signal we chose. In a real system, choosing the wrong target or loss can optimise the wrong behaviour very efficiently.

Step 2 — Put the forward pass on a computation graph

A computation graphComputation graphA diagram or data structure that represents a calculation as connected operations and values. It makes clear what was calculated in the forward pass and provides the paths used to apply the chain rule backwards. makes the calculation explicit as connected values and operations. Our graph has three operations: multiply, subtract, and square.

The forward pass goes left to right. Backpropagation goes right to left. It does not reverse the model’s answer. It calculates how a small change in an earlier value would change the final loss.

Step 3 — Use the chain rule, one local slope at a time

A derivativeDerivativeA number describing how a small change in one value changes another value nearby. In model training, a derivative of loss with respect to a parameter tells whether increasing that parameter would locally increase or decrease loss, and by how much. is a local rate of change. For squared error, the derivative of error² with respect to error is 2 × error. At error = −1, that is −2.

We need the derivative of loss L with respect to weight w. The chain rule says multiply the local effects along the path:

dL/dw = dL/derror × derror/dy × dy/dw

For our example:

dL/derror = 2 × error = −2
derror/dy = 1
dy/dw = x = 2

dL/dw = −2 × 1 × 2 = −4

dL/dw = −4 means a small increase in w would locally decrease loss. The sign matters. A gradient is not “the amount to subtract”; it points uphill. To reduce loss, gradient descent goes in the opposite direction.

Worked example — make one update and check it

Gradient descentGradient descentAn optimisation procedure that repeatedly changes parameters in the direction opposite the loss gradient. The gradient supplies a local direction; practical training also needs a learning rate, batches, stability checks, and validation evidence. updates a parameter using a learning rateLearning rateA positive setting that controls the size of an optimiser’s parameter update. Too large can overshoot or diverge; too small can make training impractically slow. It must be evaluated with the full training and validation process., a positive step-size setting:

new_weight = old_weight − learning_rate × gradient

With learning rate 0.1 and gradient −4:

new_weight = 0 − 0.1 × (−4)
new_weight = 0.4

Now run a new forward pass, using w = 0.4:

new prediction = 0.4 × 2 = 0.8
new error = 0.8 − 1 = −0.2
new loss = (−0.2)² = 0.04

Loss fell from 1 to 0.04 on this example. That is a useful sanity check, not a generalisation result. A real training step averages or sums loss over a batch, repeats across training data, and still needs validation data to reveal whether the improvement survives unseen examples.

Try it — manual forward, backward, and update in Python

x = 2.0
target = 1.0
weight = 0.0
learning_rate = 0.1

# Forward pass
prediction = weight * x
error = prediction - target
loss_before = error ** 2

# Backward pass: local derivatives multiplied by the chain rule
d_loss_d_error = 2 * error
d_error_d_prediction = 1.0
d_prediction_d_weight = x
gradient = d_loss_d_error * d_error_d_prediction * d_prediction_d_weight

# Optimiser step
new_weight = weight - learning_rate * gradient
new_loss = (new_weight * x - target) ** 2

print('loss before:', loss_before)
print('dL/dw:', gradient)
print('new weight:', new_weight)
print('loss after:', new_loss)

Step 4 — Check a gradient before trusting it

A small automatic differentiationAutomatic differentiationA technique used by machine-learning frameworks to calculate exact derivatives of a program’s composed operations, up to floating-point precision. It is not numerical approximation by tiny input changes and it is not symbolic algebra; frameworks commonly implement reverse-mode automatic differentiation for neural-network training. system calculates chain-rule gradients for framework operations. It is exact for those operations up to normal floating-point effects. It cannot detect a wrong target, incorrect data, an unsuitable loss, a missing mask, or a model that is unsafe to deploy.

For a small debugging check, estimate a slope with two nearby forward passes. This finite difference is slower and approximate, so it is for checking—not for training a large network.

def loss_for(weight: float) -> float:
    prediction = weight * 2.0
    return (prediction - 1.0) ** 2


weight = 0.0
epsilon = 0.0001

numerical = (loss_for(weight + epsilon) - loss_for(weight - epsilon)) / (2 * epsilon)
manual = 2 * (weight * 2.0 - 1.0) * 2.0

print('numerical:', round(numerical, 4))
print('manual:', round(manual, 4))

If these disagree for a small test, stop and inspect the calculation before launching a long training run. In larger systems, test a few parameters and keep tolerances realistic because floating-point arithmetic and very small epsilon values can introduce numerical noise.

Real-world AI engineering example — a training run that starts to fail

Common mistakes

Knowledge check

Practice activity

x = 3.0
target = 1.0
weight = 0.0
learning_rate = 0.1

prediction = weight * x
error = prediction - target
loss = error ** 2

gradient = 2 * error * x
new_weight = weight - learning_rate * gradient
new_loss = (new_weight * x - target) ** 2

print('loss:', loss)
print('gradient:', gradient)
print('new weight:', new_weight)
print('new loss:', new_loss)

Key takeaways

Flashcards

One-page sketchnote summary

forward → loss
backward → gradient
parameter ← parameter − rate × gradient

Resources