Linear & logistic regression

Linear regression predicts a number with a weighted sum.

Logistic regression uses the same kind of weighted sum, but interprets it as log odds and converts it into a probability.

Why this matters

Linear and logistic regression are strong baselines, fast to fit, inspectable, and widely used inside real systems.

They also expose the mechanics that more complex models reuse: weighted features, a loss function, gradients, probabilities, thresholds, and honest evaluation.

Plain-English mental model

Part 1 — A linear model adds weighted evidence

For one feature:

prediction = weight × feature + bias
y_hat = w×x + b

y_hat, read “y hat”, is the prediction.

• x: feature;
• w: weight or slope;
• b: bias or intercept;
• y: observed target.

If:

predicted resolution minutes = 3 × message_length + 8

then each one-unit increase in the chosen message-length unit adds three predicted minutes, while other model inputs are absent.

The intercept is the prediction at x=0. It may not be meaningful if zero lies outside the relevant data range.

Multiple linear regression

With several features:

y_hat = w1×x1 + w2×x2 + ... + wn×xn + b

Vector notation:

y_hat = w · x + b

This is still linear in the parameters. A transformed feature such as message_length² can create a curved relationship in the original input while remaining a linear model over the transformed features.

Part 2 — Residuals show what the line missed

A residualResidualThe signed difference between an observed target and a model prediction, commonly written observed minus predicted. Residual patterns help reveal bias, non-linearity, unequal variance, and unusual cases. is commonly:

residual = observed - predicted
e = y - y_hat

• positive: observation lies above the prediction;
• negative: observation lies below;
• zero: exact prediction.

hours = [1, 2, 3]
observed = [3, 5, 8]
weight = 2
bias = 1

predicted = [weight * x + bias for x in hours]
residuals = [
    y - y_hat
    for y, y_hat in zip(observed, predicted)
]

print("predicted:", predicted)
print("residuals:", residuals)

Residuals are not merely one final score. Plot them against predictions, features, groups, and time. Patterns reveal what the model systematically misses.

Part 3 — Least squares chooses a line

Positive and negative residuals can cancel. Absolute values avoid cancellation but have a sharp point at zero.

Least squaresLeast squaresA fitting objective that chooses parameters to minimise the sum or mean of squared residuals. Squaring prevents cancellation and gives large errors greater influence. squares every residual:

sum of squared errors = Σ(y - y_hat)²
mean squared error = (1/n) × Σ(y - y_hat)²

Squaring:

• makes every contribution non-negative;
• gives large errors more influence;
• creates a smooth objective with a convenient derivative.

It also makes the fit sensitive to outliers.

Ordinary least squares

Ordinary least squares, shortened to OLS, is the standard unweighted least-squares fit.

For one feature, the fitted slope has a closed form:

w = Σ((x - mean_x)(y - mean_y)) / Σ((x - mean_x)²)
b = mean_y - w×mean_x

For multiple features, linear algebra solves the system. Production libraries use numerically stable decompositions rather than explicitly calculating a matrix inverse.

Gradient descent is also useful, especially for large datasets or models that add other differentiable terms.

Part 4 — One gradient update by hand

Suppose there is one example:

x = 2
y = 5
w = 1
b = 0

Prediction:

y_hat = 1×2 + 0 = 2

Use squared loss:

L = (y_hat - y)² = (2 - 5)² = 9

The derivatives are:

dL/dw = 2(y_hat-y)×x
       = 2(-3)×2
       = -12

dL/db = 2(y_hat-y)
       = -6

With learning rate 0.1:

w_new = 1 - 0.1×(-12) = 2.2
b_new = 0 - 0.1×(-6)  = 0.6

New prediction:

2.2×2 + 0.6 = 5

This one-example step lands exactly on the target. Across many examples, gradients are aggregated and steps rarely solve everything at once.

Expected output:

weight: 2.0
bias: 1.0
x=5 prediction: 11.0

Part 5 — Feature scale changes optimisation

Suppose one feature ranges from 0 to 1 and another from 0 to 1,000,000.

Their gradients can have very different magnitudes. Gradient descent may zigzag or require a tiny learning rate.

Standardisation transforms a feature:

scaled_x = (x - training_mean) / training_standard_deviation

Fit the mean and standard deviation on training data only.

For plain OLS without regularisation, scaling changes coefficient units but not the best possible predictions when handled consistently. It can still improve numerical conditioning. With regularisation or gradient optimisation, scaling often materially affects the fit.

Part 6 — Interpret coefficients carefully

In multiple linear regression:

Holding the other included features fixed, a one-unit increase in x_j changes the predicted target by w_j.

Important limitations:

• the unit must be stated;
• “holding others fixed” may describe an unrealistic comparison;
• correlated features make individual coefficients unstable;
• omitted confounders can distort associations;
• a predictive coefficient is not automatically causal;
• extrapolation beyond observed data can be unreliable.

If training_hours and job_level are strongly related, the training-hours coefficient is conditional on job level. It is not the total causal effect of giving someone another hour.

Part 7 — Check linear-model assumptions through evidence

Classical OLS uncertainty calculations use assumptions such as:

• linearity: expected target is linear in included features;
• independent errors: residuals are not correlated in an unmodelled way;
• constant variance: residual spread is similar across predictions;
• no perfect multicollinearity: one feature is not an exact linear combination of others;
• error-distribution assumptions for particular small-sample intervals or tests.

Prediction can still be useful when some textbook assumptions are imperfect, but uncertainty statements and coefficient interpretation may fail.

Inspect:

• residual versus fitted plot;
• residuals over time and groups;
• extreme leverage points;
• train/validation gap;
• errors by relevant slices;
• performance under shifts.

A funnel-shaped residual plot suggests unequal variance. A curved pattern suggests missing non-linearity.

Part 8 — Linear regression failure modes

Linear regression can fail when:

• the relationship is strongly nonlinear;
• important interactions are missing;
• outliers dominate squared loss;
• categories are encoded as meaningless ordered numbers;
• features are collinear;
• target variance changes with inputs;
• predictions must obey a bound, such as non-negative counts;
• data leaks the target or future.

Possible responses include transformations, interactions, robust losses, regularisation, a different target distribution, trees, or a different model family. Diagnose before escalating complexity.

Part 9 — Why classification needs a different output

For a binary target y ∈ {0,1}, a raw line can predict values below 0 or above 1.

Clipping hides the problem and creates awkward fitting.

Logistic regression keeps the linear score:

z = w · x + b

but maps it to a probability using a sigmoid functionSigmoid functionThe S-shaped function one divided by one plus exp of negative z. It maps any real-valued logit to a value strictly between zero and one.:

p = 1 / (1 + exp(-z))

Properties:

Large positive logits approach 1. Large negative logits approach 0. They never reach either endpoint for finite logits.

Part 10 — Probability, odds, and log odds

OddsOddsProbability of an event divided by probability of it not happening, p divided by one minus p. Probability 0.75 corresponds to odds 3 to 1. are:

odds = p / (1-p)

Examples:

A logitLogitThe natural logarithm of odds, log of p divided by one minus p. Logistic regression models this unrestricted real-valued quantity as a weighted sum of features. is the natural logarithm of odds:

logit(p) = log(p / (1-p))

Logit transforms probabilities from (0,1) to all real numbers.

Logistic regression states:

log(p / (1-p)) = w · x + b

Sigmoid is the inverse transformation.

Interpret a logistic coefficient

For coefficient w_j, a one-unit feature increase:

adds w_j to log odds
multiplies odds by exp(w_j)

If w_j=0.7:

odds multiplier = exp(0.7) ≈ 2.01

It does not add 70 percentage points. Probability change depends on the starting probability.

Part 11 — Log loss fits logistic regression

For binary target y and predicted probability p:

binary cross-entropy
= -[y log(p) + (1-y) log(1-p)]

If y=1, loss is -log(p).

If y=0, loss is -log(1-p).

Confident wrong predictions receive large loss.

Expected output:

z=-2.0 p=0.119 y=0 loss=0.127
z= 0.0 p=0.500 y=1 loss=0.693
z= 2.0 p=0.881 y=1 loss=0.127

Stable libraries combine sigmoid and log loss to avoid overflow and log(0).

Part 12 — Thresholds create decisions and boundaries

A fitted probability is not yet a decision.

predicted_class = probability >= threshold

At threshold 0.5:

p ≥ 0.5
logit ≥ 0
w · x + b ≥ 0

The set where w·x+b=0 is the decision boundaryDecision boundaryThe set of feature values where a classifier changes its predicted class for a chosen threshold. Logistic regression creates a linear boundary in its input feature space unless features were transformed nonlinearly..

With two raw features, it is a line. With three, a plane. With transformed nonlinear features, the boundary can curve in original input space.

Do not default blindly to 0.5

Choose a threshold using validation evidence and operational costs.

For urgent-ticket triage:

• lower threshold catches more urgent cases but raises more false alerts;
• higher threshold reduces alerts but may miss urgent cases.

Model fitting estimates a score or probability. Product policy chooses the action trade-off.

Changing threshold does not refit weights.

Part 13 — Fit logistic regression from scratch

For logistic cross-entropy, the gradient has a clean form:

error = probability - target
gradient_weight_j = average(error × feature_j)
gradient_bias = average(error)

Expected output:

1 0.005
2 0.162
3 0.888
4 0.997
0.5 boundary: 2.443

With perfectly separable unregularised data, logistic coefficients can keep growing rather than reach a finite maximum-likelihood estimate. Production implementations use convergence checks and commonly regularisation.

Part 14 — Multiclass extensions

Binary logistic regression handles two classes.

For more classes:

• one-vs-rest: fit one binary classifier per class;
• multinomial logistic regression: produce one logit per class and use softmax.

Softmax probabilities total 1 across mutually exclusive classes.

Do not force mutually compatible labels into one multiclass target. A document that can be both “billing” and “urgent” needs multilabel treatment, often one sigmoid output per label.

Worked example — predict ticket escalation

Goal: estimate escalation probability when a ticket arrives.

1. Define the row and target

One row is a ticket snapshot at creation.

Target:

1 if escalated under policy v2 within 24 hours
0 otherwise

This target still needs auditing: escalation may reflect staffing or policy inconsistency.

2. Select features

• message-length standard score;
• number of previous contacts before creation;
• account age;
• documented issue categories encoded appropriately.

Do not include escalated_at, final agent notes, or later resolution fields.

3. Establish a baseline

Compare:

• majority-class prediction;
• documented critical-phrase rule;
• logistic regression.

4. Fit

Standardise numerical features using training statistics. Fit coefficients on training data by minimising log loss with regularisation.

5. Validate probabilities and threshold

Check:

• log loss;
• calibration;
• precision and recall;
• alert volume;
• errors by language and issue group.

Choose the operating threshold with validation data and human-review capacity.

6. Test and interpret

Open the time-separated test set after choices are frozen.

Coefficients can identify useful associations, but do not claim causes. Correlated features and selection processes affect values.

In a real AI system

Common mistakes

Exercises

Knowledge check

Practice activity

from math import exp

x = 6

linear_prediction = 2.5 * x + 4
observed = 21
residual = observed - linear_prediction

logit = -4 + 0.9 * x
probability = 1 / (1 + exp(-logit))
odds = probability / (1 - probability)

print("prediction:", linear_prediction)
print("residual:", residual)
print("squared residual:", residual ** 2)
print("logit:", round(logit, 3))
print("probability:", round(probability, 3))
print("odds:", round(odds, 3))
print("class at 0.5:", int(probability >= 0.5))
print("class at 0.9:", int(probability >= 0.9))

prediction: 19.0
residual: 2.0
squared residual: 4.0
logit: 1.4
probability: 0.802
odds: 4.055
class at 0.5: 1
class at 0.9: 0

target logit = log(0.7/0.3) ≈ 0.847
-4 + 0.9x = 0.847
x ≈ 5.386

Key takeaways

Flashcards

One-page sketchnote summary

linear = weight * x + bias
residual = observed - linear
logit = weight * x + bias
probability = 1 / (1 + exp(-logit))
decision = probability >= threshold

Resources