From neurons to networks

An app needs to spot support tickets that may lock a customer out. A single rule such as “three failed sign-ins means urgent” is easy to write. Real cases combine signals: recent account creation, location change, recovery attempts, and the pattern across them.

A neural network is a way to learn combinations of numeric signals. It is not magic, a database, or a mind. It is many small calculations with adjustable numbers.

Why this matters

Neural networks power image recognition, speech systems, recommendation, and language models. Before using a framework, you need to know what a layer calculates. That makes model shapes, unexpected outputs, and later training failures less mysterious.

Plain-English mental model

Step 1 — A perceptron makes a weighted decision

An input is a number supplied to the model. For a ticket, it could be the number of failed sign-ins. A weight is a learned number that scales an input’s contribution. A bias is an extra learned baseline that lets the calculation shift before it reacts to inputs.

First calculate the weighted sum, often written as z:

z = x₁w₁ + x₂w₂ + ... + b

Here x means an input, w means its weight, and b means bias. This is a dot product plus a bias, which you met in the maths foundations.

Small example: use x₁=4 failed sign-ins, x₂=2 days of account age, weights 0.8 and −0.4, and bias −1.

z = (4 × 0.8) + (2 × −0.4) − 1
z = 3.2 − 0.8 − 1
z = 1.4

The negative weight does not mean account age is “bad”. It means that, for the training objective and other available inputs, a higher value moves this unit’s calculation down. Do not turn one weight into a story about cause or policy.

Step 2 — An activation function adds a bend

If we only stack weighted sums, the result is still one big linear calculation. A linear rule can draw a straight decision boundary. It cannot express many useful “if this pattern, then that pattern” shapes.

An activation functionActivation functionA function applied after a neuron’s weighted sum. It adds non-linearity, meaning a network can represent curved and conditional relationships instead of only one straight-line transformation. transforms z after the weighted sum. It adds non-linearity: a bend or gate that lets stacked layers represent more varied relationships.

The beginner-friendly example is ReLUReLURectified Linear Unit, an activation function defined as max(0, input). It outputs zero for negative values and keeps positive values unchanged. It is simple and commonly used, but a unit can stop contributing if it stays negative., short for Rectified Linear Unit:

ReLU(z) = max(0, z)

So ReLU(1.4) is 1.4, while ReLU(−0.7) is 0. This does not mean negative evidence is always useless. It is a design choice for that particular hidden value.

GELUGELUGaussian Error Linear Unit, a smooth activation function that gradually gates values rather than using ReLU’s sharp zero corner. It is common in transformer-style networks. is a smooth gate used in many transformer networks. SwiGLUSwiGLUA gated activation used in many modern language-model feed-forward layers. It transforms one path for values, uses another transformed path as a learned gate, and multiplies them. The exact formula is less important than recognising it as a flexible gated non-linearity. is a gated activation: one learned path produces values and another controls how much of those values pass through. You do not need to memorise their formulas yet. Remember the job: preserve useful signal while giving the network a non-linear way to combine it.

Step 3 — Layers create intermediate representations

A hidden layerHidden layerA layer of calculations between a network’s visible input and output. Its intermediate values are called hidden because they are internal model representations, not because they are secret or unknowable. sits between the visible input and output. “Hidden” only means internal to the model. Its values can be inspected, although they may not have a simple human label such as “fraud risk”.

For the ticket example, one hidden unit might react to a suspicious combination of failures and a young account. Another could react to a different combination. An output layer combines those intermediate values into a score.

A forward passForward passOne calculation that moves an input through a model’s current parameters to produce an output. A forward pass does not itself learn or change the parameters; backpropagation and an optimiser do that during training. is one trip from input to output using the current values of all parameters. A parameter is any learned number in the model, including weights and biases. A forward pass predicts; it does not learn. The next lesson explains how backpropagation measures which parameters contributed to an error and how an optimiser updates them.

Worked example — two neurons, one output

Suppose the first hidden unit has output h₁ = ReLU(1.4) = 1.4. A second unit receives a negative sum and has h₂ = ReLU(−0.7) = 0.

The output layer uses weights 1.2 and 0.3, with no output bias for this tiny example:

score = (1.2 × h₁) + (0.3 × h₂)
score = (1.2 × 1.4) + (0.3 × 0)
score = 1.68

1.68 is a score, not automatically a probability or an instruction. A real system decides how to transform a score, choose a threshold, request more evidence, or hand the case to a person. That policy must be validated for errors, workload, privacy, and fairness.

Try it — write one perceptron in Python

def relu(value: float) -> float:
    return max(0.0, value)


failed_sign_ins = 4.0
account_age_days = 2.0

weighted_sum = (
    failed_sign_ins * 0.8
    + account_age_days * -0.4
    - 1.0  # bias
)

hidden_value = relu(weighted_sum)
urgency_score = hidden_value * 1.2

print("weighted sum:", weighted_sum)
print("hidden value:", hidden_value)
print("urgency score:", urgency_score)

The long decimal is normal floating-point representation, not evidence that the calculation is conceptually wrong. For display, use round(value, 2). Do not round values during training just to make logs tidier.

Modification ideas: change one input at a time; set weighted_sum negative; add a second hidden unit; then compare what changes. Do not claim the output is a calibrated risk probability unless the full model was trained and evaluated for that purpose.

Step 4 — Starting values matter before learning

Training begins with initialisationInitialisationThe choice of starting values for model parameters before training. Good initialisation keeps signals and gradients at practical sizes; it does not replace training or guarantee quality.: choosing starting parameter values. If every neuron starts with the same values, equivalent neurons can keep doing the same work. If values are wildly large or tiny, signals and later gradients can become unstable.

Common initialisation methods choose small random values scaled for the layer shape. The important beginner rule is: use the framework default unless you have a measured reason to change it. Random starts mean training results can vary slightly, so record seeds and compare repeat runs when reliability matters.

You may hear universal approximationUniversal approximationThe mathematical result that a sufficiently large neural network with a suitable activation can approximate many continuous functions on a bounded input range. It does not say that the network will learn the function from limited data, train efficiently, generalise, or be the best design.: sufficiently large networks can approximate many functions over a bounded range. This is a possibility result, not a product plan. It does not promise enough data, easy training, generalisation, low cost, safety, or a good user experience.

Real-world AI engineering example — triaging support safely

Common mistakes

Knowledge check

Practice activity

def relu(value):
    return max(0.0, value)


failed_sign_ins = 4.0
account_age_days = 2.0
bias = -1.0

z = failed_sign_ins * 0.8 + account_age_days * -0.4 + bias
hidden = relu(z)
score = hidden * 1.2

print('z:', z)
print('hidden:', hidden)
print('score:', score)

Key takeaways

Flashcards

One-page sketchnote summary

z = Σ(input × weight) + bias
hidden = ReLU(z)
score = Σ(hidden × output_weight)

Resources