Maths I: linear algebra & calculus

AI mathematics starts with two practical questions:

1. How can we represent many related numbers?
2. How should those numbers change when the model is wrong?

Linear algebra answers the first question with vectors and matrices. Calculus answers the second with derivatives and gradients.

Why this matters

Embeddings are vectors. Neural-network weights are matrices. Attention uses matrix multiplication and dot products. Training uses gradients and the chain rule.

You do not need to become a mathematician before building AI systems. You do need a dependable mental model for shapes, similarity, transformations, and change.

The intuition

Part 1 — Vectors package related numbers

A vectorVectorAn ordered list of numbers that can represent a point, a direction, or a collection of features. In AI engineering, vectors represent items such as model inputs, parameters, and embeddings; their order and dimension must have a defined meaning. is an ordered list of numbers.

ticket features = [message_length, number_of_replies, customer_age_days]
                = [120,            3,                 45]

Order matters. [120, 3, 45] is useful only when every position has an agreed meaning.

The dimension of a vector is its number of entries. This ticket vector has dimension 3.

A vector can be understood in several compatible ways:

• a list of features;
• a point in a coordinate space;
• an arrow from the origin to that point;
• a direction and a magnitude.

In code, shape records how many values exist along each axis. A vector with three entries has shape (3,).

Add and scale vectors

Vectors of the same dimension add entry by entry:

[2, 1] + [3, 4] = [5, 5]

Multiplying by a scalar—one ordinary number—scales every entry:

3 × [2, 1] = [6, 3]

Geometrically, positive scaling changes arrow length without changing its direction. A negative scale reverses direction.

Norms measure size

A norm turns a vector into a non-negative measure of size. The most familiar is the Euclidean, or L2, norm:

norm([3, 4]) = square_root(3² + 4²) = 5

That is the Pythagorean distance from the origin.

Other norms answer different questions:

• L1 norm adds absolute values: |3| + |4| = 7.
• L2 norm uses the square root of squared values: 5.
• Maximum norm uses the largest absolute entry: 4.

The “right” norm depends on the problem. In AI, L2 norms appear in distance, normalisation, regularisation, and gradient clipping.

Part 2 — Dot products produce one alignment score

The dot productDot productA single number produced by multiplying corresponding vector entries and adding the products. It measures weighted alignment and is the core calculation inside matrix multiplication, neural layers, attention scores, and projections. multiplies corresponding entries and adds the results.

[2, 1, 3] · [4, -2, 5]
= 2×4 + 1×(-2) + 3×5
= 8 - 2 + 15
= 21

The input is two vectors of equal dimension. The output is one scalar.

def dot(left: list[float], right: list[float]) -> float:
    if len(left) != len(right):
        raise ValueError("vectors must have the same dimension")

    products = [a * b for a, b in zip(left, right)]
    return sum(products)


score = dot([2, 1, 3], [4, -2, 5])
print(score)

Why is this useful? Suppose features = [urgency, customer_value, age] and weights = [2.0, 0.5, -0.1]. Their dot product is a weighted score. Positive weights increase the output; negative weights reduce it.

Geometrically:

a · b = norm(a) × norm(b) × cosine(angle)

The dot product grows when vectors are long and point in similar directions. It is zero for perpendicular non-zero vectors and negative when they point more oppositely than similarly.

Part 3 — Cosine similarity compares direction

Cosine similarityCosine similarityA comparison of two vectors based on the cosine of the angle between them. It ranges from minus one to one for non-zero real vectors and focuses on direction rather than magnitude, which often makes it useful for comparing embedding meaning. divides the dot product by both vector lengths:

cosine_similarity(a, b) = (a · b) / (norm(a) × norm(b))

For non-zero real vectors, the result lies between -1 and 1:

• 1: same direction;
• 0: perpendicular directions;
• -1: opposite directions.

Cosine similarity ignores overall scale. [1, 2] and [10, 20] have similarity 1 because they point in the same direction.

That can be useful for embeddings, where direction often carries more semantic information than length. It can also discard meaningful magnitude. If vector length represents confidence or quantity, cosine similarity may hide an important difference.

Never calculate cosine similarity with a zero vector. Its norm is zero, so the denominator is zero and the angle has no defined direction.

Expected output:

login similarity:    0.995
delivery similarity: 0.293

Part 4 — Matrices organise transformations

A matrix is a rectangular grid of numbers.

A = [[1, 2, 0],
     [0, 1, 3]]

This matrix has 2 rows and 3 columns, so its shape is (2, 3).

A matrix can represent:

• a batch of row vectors;
• a table of features;
• the weights of a neural layer;
• a linear transformation from one vector space to another.

The same numbers can have different meanings depending on the convention. Always state shape and axis meaning.

Matrix-vector multiplication

Multiply each matrix row by the vector using a dot product:

[[1, 2, 0],    [4]    [1×4 + 2×5 + 0×6]    [14]
 [0, 1, 3]] ×  [5] =  [0×4 + 1×5 + 3×6] =  [23]
                [6]

A (2, 3) matrix maps a dimension-3 input to a dimension-2 output.

This is the core of a neural layer:

output = weights × input + bias

The layer mixes input features according to learned weights, then adds one bias per output feature.

Matrix-matrix multiplication

Matrix multiplicationMatrix multiplicationAn operation that combines rows from one matrix with columns from another through dot products. It composes linear transformations and mixes input features into output features; the inner dimensions must match. applies the same idea to every row-column pair.

Shapes make the rule visible:

(m, n) × (n, p) → (m, p)

The inner dimensions must match because each dot product needs equal-length vectors. The outer dimensions become the output shape.

Matrix multiplication is not entry-by-entry multiplication. It is also usually not commutative:

A × B is generally different from B × A

Sometimes the reverse product is not even shape-compatible.

Expected output:

row 0 · column 0 = 14
row 0 · column 1 = 1
row 1 · column 0 = 23
row 1 · column 1 = 6
result: [[14, 1], [23, 6]]

Part 5 — Projection, rank, eigenvectors, and SVD

These ideas sound abstract because their names arrive before their jobs. Start with the jobs.

Projection asks “how much lies in this direction?”

Projecting vector a onto non-zero vector b produces the part of a that points along b.

projection_of_a_onto_b = ((a · b) / (b · b)) × b

The dot-product ratio decides how much of direction b is present. Projection appears in least squares, dimensionality reduction, and decomposing gradients.

Example:

a = [3, 4]
b = [1, 0]

projection = 3 × [1, 0] = [3, 0]

The projection keeps the horizontal part and discards the vertical part.

Rank asks “how many independent directions survive?”

The rank of a matrix is the number of linearly independent directions in its rows or columns.

Linearly independent means no direction can be built by scaling and adding the others.

[[1, 2],
 [2, 4]]

The second row is exactly twice the first. Although the matrix has two rows, it carries only one independent row direction. Its rank is 1.

Rank tells you how much information a linear transformation can preserve. A low-rank transformation compresses many input possibilities into fewer independent output directions.

Eigenvectors ask “which directions keep their direction?”

For a square matrix A, an eigenvector v is a non-zero direction that the matrix only scales:

A × v = λ × v

The Greek letter lambda, λ, is the eigenvalue: the scale factor.

Most vectors rotate or change direction under a transformation. Eigenvectors reveal the special directions that do not.

They help analyse repeated transformations, covariance, dynamics, and principal components. Not every real matrix has a full set of real eigenvectors, so treat this as a specific tool rather than a universal decomposition.

SVD asks “what are the main input and output directions?”

Singular value decomposition, shortened to SVD, factorises any real matrix:

A = U × Σ × Vᵀ

Read it as three stages:

1. Vᵀ rotates input into important input directions.
2. Σ scales those directions by singular values.
3. U rotates them into output directions.

Large singular values identify directions the transformation preserves strongly. Small values identify weak directions. Keeping only the largest values gives a lower-rank approximation.

This supports compression, denoising, latent semantic analysis, principal component analysis, and low-rank model adaptation.

Optional advanced note: for a symmetric positive-semidefinite matrix such as a covariance matrix, the SVD and eigendecomposition are closely related. You do not need that result yet.

Part 6 — Derivatives measure local change

A function maps inputs to outputs. A derivative measures how quickly one scalar output changes as one scalar input changes.

For:

f(x) = x²

the derivative is:

f′(x) = 2x

At x = 3, the derivative is 6. Near 3, increasing x by a small amount such as 0.01 increases x² by roughly 6 × 0.01 = 0.06.

The derivative is local. It describes the tangent slope at one point, not the whole curve.

Estimate a derivative numerically

The finite-difference approximation is:

f′(x) ≈ (f(x + h) - f(x)) / h

where h is a small positive step.

def derivative(function, x: float, h: float = 0.0001) -> float:
    return (function(x + h) - function(x)) / h


print(round(derivative(lambda x: x**2, 3.0), 3))

Expected output is approximately 6.0.

If h is too large, the estimate is not local enough. If h is extremely small, floating-point rounding can dominate. Automatic differentiation avoids choosing h; later deep-learning lessons explain it.

Part 7 — Partial derivatives and gradients

When a function has several inputs, a partial derivative changes one input while holding the others fixed.

For:

loss(w, b) = (w + b - 5)²

at w=1, b=1, the error is -3.

partial loss / partial w = 2(w + b - 5) = -6
partial loss / partial b = 2(w + b - 5) = -6

The gradientGradientA vector containing the partial derivative of a scalar output with respect to each input variable. It points in the local direction of steepest increase, so gradient descent subtracts a scaled gradient to reduce a loss. collects these partial derivatives:

gradient = [-6, -6]

It has the same shape as the inputs [w, b]. The gradient points toward the steepest local increase. To reduce loss, gradient descent subtracts it:

new_parameter = old_parameter - learning_rate × gradient

The learning rate controls step size:

• Too small: learning may be extremely slow.
• Too large: updates may overshoot or diverge.
• Appropriate: loss usually decreases, though noisy batches may not decrease every step.

Gradient descent finds a local path, not a guaranteed global best solution for a neural network.

Part 8 — The chain rule connects composed operations

Models are compositions: one operation feeds the next.

x → multiply by w → add b → square error → loss

The chain ruleChain ruleA calculus rule for differentiating a composition of functions. It multiplies local rates of change along the path from an input to an output, which is how backpropagation assigns credit through many neural-network operations. says that the total rate of change is the product of local rates along the path.

Worked example:

prediction = w × x
loss = (prediction - target)²

Use x=3, w=2, and target=10.

Forward pass:

prediction = 2 × 3 = 6
error = 6 - 10 = -4
loss = (-4)² = 16

Backward pass:

change of loss with error      = 2 × error = -8
change of error with prediction = 1
change of prediction with w     = x = 3

change of loss with w = -8 × 1 × 3 = -24

The negative derivative says increasing w locally reduces loss. With learning rate 0.01:

new w = 2 - 0.01 × (-24) = 2.24

New prediction is 2.24 × 3 = 6.72, closer to 10.

This local multiplication is the heart of backpropagation. Each operation needs the gradient arriving from later computation and its own local derivative.

Part 9 — Jacobians track several outputs

A gradient covers one scalar output with respect to many inputs.

A JacobianJacobianA matrix of all first-order partial derivatives for a vector-valued function. Each row tracks how one output changes with every input, generalising a gradient from one scalar output to several outputs. covers many outputs with respect to many inputs. It is a matrix:

inputs:  [x₁, x₂]       two inputs
outputs: [y₁, y₂, y₃]   three outputs

Jacobian shape: (3, 2)

Each row answers: “How does this output change with each input?”

J = [[∂y₁/∂x₁, ∂y₁/∂x₂],
     [∂y₂/∂x₁, ∂y₂/∂x₂],
     [∂y₃/∂x₁, ∂y₃/∂x₂]]

Deep-learning libraries usually avoid constructing enormous full Jacobian matrices. Backpropagation efficiently computes a vector-Jacobian product: only the combined gradient needed for the scalar loss.

That implementation detail is optional now. Remember the shape rule and the job: a Jacobian records every first-order input-output sensitivity for a vector-valued function.

Worked example — score two support tickets

Suppose each ticket has a two-feature vector:

ticket = [billing_words, urgency_words]

A classifier has two output categories, with weight matrix:

weights = [[ 1.5, 0.2],   billing score
           [-0.3, 1.2]]   urgent score

For ticket [2, 3]:

billing score = [1.5, 0.2] · [2, 3] = 3.0 + 0.6 = 3.6
urgent score  = [-0.3, 1.2] · [2, 3] = -0.6 + 3.6 = 3.0

The matrix maps two input features to two output scores.

If the correct category is urgent but billing scores higher, a loss function measures the mistake. Calculus computes how that loss changes with each of the four weights. The gradient has the same shape as the weight matrix (2, 2). Training subtracts a scaled gradient, slightly changing all four weights.

This example links the whole lesson:

vector input
→ matrix multiplication made of dot products
→ vector scores
→ scalar loss
→ gradient through the chain rule
→ updated matrix

In a real AI system

Common mistakes

Exercises

Knowledge check

Practice activity

from math import sqrt

query = [0.9, 0.8, 0.1]
documents = {
    "Password help": [0.8, 0.7, 0.2],
    "Account settings": [0.6, 0.5, 0.4],
    "Delivery tracking": [0.1, 0.2, 0.95],
}

# Add dot, norm, cosine_similarity, and ranking.

from math import sqrt

def dot(a, b):
    if len(a) != len(b):
        raise ValueError("dimensions must match")
    return sum(x * y for x, y in zip(a, b))

def norm(vector):
    return sqrt(dot(vector, vector))

def cosine_similarity(a, b):
    denominator = norm(a) * norm(b)
    if denominator == 0:
        raise ValueError("zero vectors have no direction")
    return dot(a, b) / denominator

query = [0.9, 0.8, 0.1]
documents = {
    "Password help": [0.8, 0.7, 0.2],
    "Account settings": [0.6, 0.5, 0.4],
    "Delivery tracking": [0.1, 0.2, 0.95],
}

ranked = sorted(
    (
        (cosine_similarity(query, vector), name)
        for name, vector in documents.items()
    ),
    reverse=True,
)

for score, name in ranked:
    print(f"{score:.3f}  {name}")

0.995  Password help
0.924  Account settings
0.293  Delivery tracking

Key takeaways

Flashcards

One-page sketchnote summary

score = dot(features, weights)
parameter -= learning_rate * gradient

Resources