Bias–variance trade-off & regularisation

Generalisation means performing well on useful unseen cases, not memorising last month’s rows.

Some models are too simple to learn the pattern. Others are flexible enough to learn noise. Regularisation is one controlled way to prefer patterns that survive new data.

Why this matters

The same problem appears in classical ML, deep learning, prompts, retrieval, and agents: a system looks good on examples it has already influenced, then fails on new work.

Plain-English mental model

Step 1 — Read training and validation evidence together

Underfitting often shows high training and validation error. The representation or model cannot express a useful relationship.

Overfitting often shows low training error but noticeably worse validation error. The model is sensitive to the particular training sample.

BiasBiasError from systematic simplifying assumptions that make a model unable to represent a useful pattern. In the bias-variance trade-off it does not mean social bias, although both require careful review. means systematic approximation error from overly restrictive assumptions. VarianceVarianceSensitivity of a fitted model to which training examples it happened to see. High variance can fit training noise and produce unstable unseen-data performance. means sensitivity to training data. These technical terms are distinct from social bias, which must be assessed separately.

Step 2 — Regularisation adds a cost for unnecessary complexity

Objective:

objective = data_loss + strength × penalty(weights)

Strength is often called alpha, lambda, or C. Conventions differ: in many APIs a larger C means less regularisation. Read documentation and tune safely.

L1 and L2

L1 regularisationL1 regularisationA penalty proportional to the sum of absolute parameter values. It can set some linear-model weights exactly to zero, but correlated features can make its selections unstable. adds absolute values:

L1 penalty = Σ |weight_j|

It can make some weights exactly zero. Correlated features can swap importance, so zero weight is not proof that a feature is irrelevant.

L2 regularisationL2 regularisationA penalty proportional to the sum of squared parameter values. It shrinks weights smoothly and is also called ridge regularisation or weight decay in some contexts. adds squares:

L2 penalty = Σ weight_j²

It discourages large individual weights and is commonly called ridge regularisation. Elastic netElastic netA regularisation method combining L1 and L2 penalties, balancing sparsity with smooth weight shrinkage through validated hyperparameters. combines L1 and L2; tune its overall strength and mix on held-out evidence.

weights = [3.0, -1.0, 2.0]
strength = 0.1

l1 = sum(abs(weight) for weight in weights)
l2 = sum(weight ** 2 for weight in weights)

print("L1 sum:", l1)
print("L2 sum:", l2)
print("L1 contribution:", strength * l1)
print("L2 contribution:", strength * l2)

Step 3 — Class imbalance is a decision problem

Class imbalanceClass imbalanceA classification dataset where one class is much rarer than another. It makes accuracy misleading and requires attention to splits, metrics, thresholds, and the data-generating process. means positives are much rarer than negatives. A 99% accurate model can predict negative for every row and still miss every positive.

Start by checking whether rarity reflects reality, labels, collection gaps, or a changing policy. Use grouped/time-aware splits and inspect prevalence in every split.

• class weights: increase loss for errors on rare classes;
• threshold tuning: choose action volume and error trade-off on validation;
• undersampling: remove majority rows and potentially lose information;
• oversampling: repeat minority rows and potentially overfit;
• SMOTE: create synthetic minority rows between nearby minority examples.

SMOTESMOTESynthetic Minority Over-sampling Technique, which creates synthetic minority examples between nearby minority rows. It belongs inside training folds only and can create unrealistic data if used blindly. must run only inside each training fold. Creating synthetic rows before splitting lets related rows enter validation or test data and falsely improve measurements. It can also create implausible examples across categorical or constrained features.

Step 4 — Tune with a declared budget and one objective

Hyperparameter tuningHyperparameter tuningSelecting settings chosen before fitting, such as regularisation strength, tree depth, or learning rate, using validation or cross-validation evidence within a declared search budget. searches choices such as regularisation strength, tree depth, neighbours, or class weights.

• Grid search: planned combinations; clear but wasteful in many dimensions.
• Random search: samples ranges; often covers influential dimensions better within a budget.
• Bayesian optimisation: selects later trials using earlier results; it can be efficient but still overfits validation.
• Optuna: a Python library that can run and prune declared searches; it does not replace sound splits or a final test.

Declare metric, folds, budget, seed policy, ranges, and stopping rule before examining results. Record every trial. If many people repeatedly tune against one validation set, use nested cross-validation or a new validation period.

Worked system example — rare incident escalation

An operations team wants to escalate rare infrastructure incidents.

1. Train on earlier weeks; preserve whole incidents and later weeks for validation and test.
2. Compare a simple baseline, class-weighted model, and threshold policy.
3. Measure PR-AUC plus precision and recall at the human team’s daily capacity.
4. Fit scalers and resampling inside each training fold.
5. Review misses and false alarms with operators, including service and region slices.
6. Select a documented threshold on validation, then test once on a later period.

The aim is not the lowest average loss. It is a safe operational policy with an honest fallback.

Common mistakes

Exercises

Knowledge check

Practice activity

candidates = {
    "A simpler": {"train_error": 0.16, "validation_error": 0.17},
    "B flexible": {"train_error": 0.03, "validation_error": 0.25},
}

for name, result in candidates.items():
    gap = result["validation_error"] - result["train_error"]
    print(name, "gap:", round(gap, 2))

print("review capacity:", 25, "alerts per day")

A simpler gap: 0.01
B flexible gap: 0.22
review capacity: 25 alerts per day

Key takeaways

Flashcards

One-page sketchnote summary

objective = loss + strength × penalty · fit training folds · choose validation settings · report once on test

Resources