Evaluation done properly

An evaluation is a rehearsal for the future decision your system will make.

If future data leaks into training, if the metric hides the expensive mistake, or if you repeatedly peek at the test set, the rehearsal flatters the model. Production then surprises you.

Why this matters

Every later AI system needs this discipline: a prompt, retrieval system, classifier, or agent is only useful if it performs well on relevant unseen work.

Plain-English mental model

Step 1 — Define the decision before the metric

Write down:

• who or what receives a prediction;
• the decision that follows;
• the time when information is available;
• which errors are harmful, and to whom;
• capacity limits, such as human reviews per day;
• the baseline you need to beat;
• slices that must be reviewed separately.

For fraud screening, a false negative might lose money. A false positive might block a legitimate customer. “Maximise F1” is not a decision policy until those consequences are considered.

Step 2 — Protect train, validation, and test evidence

The train-validation-test splitTrain-validation-test splitA division of data into a training portion for fitting, a validation portion for model and threshold choices, and a protected test portion for a final honest estimate after choices are finished. has three jobs:

Fit every learned transform on training rows only. Means, feature selection, PCA, encoders, text vocabulary, and imputation can leak information if calculated across future validation or test rows.

Match the split to deployment

Random independent rows can be suitable only when rows are genuinely exchangeable.

• Grouped data: keep one customer, patient, document, or device in one split.
• Time-ordered data: train on earlier periods and validate/test on later periods.
• Rare classes: use stratification where it does not violate time or groups.
• Repeated decisions: ensure features exist at the intended prediction time.

Step 3 — Cross-validation uses training data more efficiently

Cross-validationCross-validationRepeatedly fitting on part of available training data and measuring on a different held-out part, so each row can serve as validation once. It helps compare choices but does not replace a final untouched test set. partitions available training data into folds. In five-fold cross-validation, fit on four folds and validate on the fifth; rotate until every fold has been held out once.

Report the distribution across folds, not just the best score. Put all learned preprocessing inside the fold pipeline. A scaler fitted before folds has already seen held-out validation rows.

Use time-series or grouped variants when random folds violate the real decision boundary. Keep a final untouched test period or group for the final report.

Step 4 — Start with the confusion matrix

For a binary decision at one threshold, the confusion matrixConfusion matrixA table of classification outcomes: true positives, false positives, false negatives, and true negatives. It makes the kinds of correct and incorrect decisions visible at one chosen threshold. counts:

• true positive (TP): predicted positive and actually positive;
• false positive (FP): positive alert on a negative case;
• false negative (FN): missed positive case;
• true negative (TN): correctly predicted negative.

Calculate core classification metrics

AccuracyAccuracyThe fraction of all predictions that are correct. It can look excellent on an imbalanced problem while the model misses nearly every rare but important positive case.:

(TP + TN) / (TP + FP + FN + TN)

PrecisionPrecisionOf predictions marked positive, the fraction that are truly positive. Precision answers whether an alert or positive decision is usually right. answers: “When we alert, how often are we right?”

TP / (TP + FP)

RecallRecallOf examples that are truly positive, the fraction the model finds. Recall answers how many real positive cases are caught., also called sensitivity, answers: “Of actual positives, how many did we find?”

TP / (TP + FN)

F1F1 scoreThe harmonic mean of precision and recall. It is useful when both matter, but it does not encode the real cost of false alarms versus missed cases. is the harmonic mean of precision and recall:

2 × precision × recall / (precision + recall)

F1 gives neither error type a monetary, safety, or fairness cost. Use it only when that balance is a reasonable proxy.

Worked example — a rare screening decision

From 1,000 accounts, 45 are truly risky. A threshold produces TP=36, FP=12, FN=9, TN=943.

accuracy  = (36 + 943) / 1000 = 0.979
precision = 36 / (36 + 12)     = 0.750
recall    = 36 / (36 + 9)      = 0.800
F1        ≈ 0.774

Accuracy looks almost perfect because negatives are common. It does not show nine missed risky accounts or twelve unnecessary interventions.

Expected output:

accuracy: 0.979
precision: 0.750
recall: 0.800
f1: 0.774

Step 5 — Thresholds turn scores into actions

Many classifiers output a score. A threshold turns it into an alert:

alert if score >= threshold

Lower threshold: more alerts, usually higher recall and lower precision.

Higher threshold: fewer alerts, usually higher precision and lower recall.

Choose on validation data using costs, review capacity, safety, fairness, and outcomes. Freeze it before the test report. A threshold can also vary only when a lawful, fair, and carefully justified policy supports it; never casually “fix” a subgroup metric with different rules.

Step 6 — Ranking curves answer a different question

The ROC-AUCROC-AUCArea under the receiver operating characteristic curve, summarising how often a scoring model ranks a random positive above a random negative across thresholds. It can look optimistic when positives are very rare. measures ranking quality across thresholds: how often a random positive receives a higher score than a random negative. It is useful but can look strong when negatives vastly outnumber positives.

PR-AUCPR-AUCArea under the precision-recall curve. It focuses on positive-class performance and is often more informative than ROC-AUC when positive cases are rare, though it still needs prevalence and threshold context. summarises the precision-recall curve. It focuses attention on the positive class, so it is often more revealing for rare fraud, defects, or incidents.

Neither curve selects an operational threshold. Always inspect:

• class prevalence;
• precision and recall at realistic alert volumes;
• confusion matrix at the chosen threshold;
• confidence intervals or variation across relevant samples;
• subgroup and time slices;
• a baseline tied to the current workflow.

Step 7 — A score is not automatically a probability

CalibrationCalibrationAgreement between predicted probabilities and observed frequencies. A well-calibrated 0.8 score corresponds to about 80% positives across comparable cases, but calibration must be checked on held-out data. asks whether predicted probabilities match observed frequencies.

If comparable cases all receive 0.80, roughly 80% should actually be positive. A model can rank very well and be badly calibrated. A score of 0.8 should not drive a risk estimate, budget, or automated action until calibration has been checked on held-out data.

Calibration can shift when prevalence, policy, or data distribution changes. Re-check it after deployment.

Step 8 — Regression has different errors

For a numeric target y and prediction ŷ:

MAEMean absolute errorThe average absolute difference between regression predictions and targets. It reports an average miss in the target's units and weights every error linearly. is average absolute miss:

MAE = mean(|y - ŷ|)

RMSERoot mean squared errorThe square root of average squared regression error. It has target units but penalises larger misses more heavily than mean absolute error. squares misses before averaging, then takes a square root:

RMSE = square_root(mean((y - ŷ)²))

RMSE penalises big mistakes more. Choose it when a large miss genuinely costs more—not because it is conventional.

MAPEMean absolute percentage errorThe average absolute regression error as a percentage of the actual value. It is undefined at zero and becomes unstable near zero, so it is often unsuitable without careful alternatives. divides error by actual value. It is undefined at actual zero and volatile near zero. Use it only when percentage error makes sense and zero is handled honestly; consider MAE, scaled errors, or a domain-specific cost instead.

actual = [10, 12, 20, 100]
predicted = [11, 10, 24, 80]

absolute_errors = [abs(y - y_hat) for y, y_hat in zip(actual, predicted)]
mae = sum(absolute_errors) / len(absolute_errors)
rmse = (sum(error ** 2 for error in absolute_errors) / len(absolute_errors)) ** 0.5

print("absolute errors:", absolute_errors)
print("MAE:", round(mae, 3))
print("RMSE:", round(rmse, 3))

Expected output:

absolute errors: [1, 2, 4, 20]
MAE: 6.75
RMSE: 10.21

The 20-unit miss matters more to RMSE because it is squared. Decide whether that reflects your real consequence.

Real-world AI engineering example

Common mistakes

Exercises

Knowledge check

Practice activity

options = {
    "lower threshold": {"tp": 18, "fp": 12, "fn": 2, "tn": 968},
    "higher threshold": {"tp": 14, "fp": 6, "fn": 6, "tn": 974},
}

for name, counts in options.items():
    tp, fp, fn = counts["tp"], counts["fp"], counts["fn"]
    precision = tp / (tp + fp)
    recall = tp / (tp + fn)
    print(name)
    print(" alerts:", tp + fp)
    print(" precision:", round(precision, 3))
    print(" recall:", round(recall, 3))

lower threshold
 alerts: 30
 precision: 0.6
 recall: 0.9
higher threshold
 alerts: 20
 precision: 0.7
 recall: 0.7

Key takeaways

Flashcards

One-page sketchnote summary

fit(train)
choose(model, threshold, validation)
report_once(test)
precision = TP / (TP + FP)
recall = TP / (TP + FN)

Resources