Data foundations

Data is the evidence an AI system learns from.

Good model code cannot rescue rows that contain the answer, mix future events into the past, identify the wrong person, or change meaning without warning.

Why this matters

A production AI system might read resumes, support tickets, product events, or medical records.

Before training or retrieval, an engineer must make those records consistent, protect personal data, preserve the correct time boundary, and make the result reproducible. Otherwise an impressive score can be evidence of a broken pipeline.

Plain-English mental model

Part 1 — Rows need meaning before code

A row usually represents one unit: one customer, ticket, message, visit, or event.

A column represents one field about that unit.

A schema describes fields and their types. A useful dataset description also states:

• what one row represents;
• how a row is uniquely identified;
• when the event occurred;
• where it came from;
• which units are included or excluded;
• how labels were produced;
• which values can be missing;
• which fields are personal or sensitive.

Start with the prediction moment:

What information is genuinely available when this system must make its decision?

That question prevents many forms of leakage later.

Part 2 — NumPy stores efficient numerical arrays

A NumPy arrayNumPy arrayA fixed-type, multidimensional block of values designed for efficient numerical operations. Its shape describes the length of each dimension, and vectorised operations can process many entries without a Python loop. is a block of values with:

• a shape: the length of each dimension;
• a dtype: the stored data type, such as 64-bit float;
• a number of dimensions, also called axes.

For a matrix with 3 rows and 2 columns:

scores.shape == (3, 2)

Unlike a Python list, one NumPy array normally uses one dtype. That regular layout enables fast numerical operations.

Vectorisation means expressing an operation over a whole array instead of writing a Python loop for each element:

import numpy as np

lengths = np.array([10.0, 20.0, 30.0])
mean = lengths.mean()
standard_deviation = lengths.std()
scaled = (lengths - mean) / standard_deviation

print(scaled.round(3))

Broadcasting, views, and copies

Broadcasting lets compatible shapes interact without manually copying a value. A scalar can be subtracted from every array entry.

Broadcasting is powerful, but a wrong shape can produce a plausible, incorrect result. Check shapes at boundaries.

A view can share the original array’s memory. A copy owns separate memory.

subset = values[:3]         # often a view
independent = values[:3].copy()

Changing a view may change the original. Use .copy() when independent data is intended.

Part 3 — pandas and Polars work with labelled tables

A DataFrameDataFrameA labelled table of rows and columns. Libraries such as pandas and Polars provide DataFrames for selecting, filtering, joining, grouping, cleaning, and summarising structured data. is a labelled table.

Use a DataFrame when column names, mixed field types, joins, grouping, dates, and missing values matter.

pandas

pandas is a mature Python library with an eager API: most operations execute when called.

import pandas as pd

tickets = pd.DataFrame({
    "channel": ["chat", "email", "chat"],
    "minutes": [4, 20, 6],
})

summary = (
    tickets
    .groupby("channel", as_index=False)
    .agg(mean_minutes=("minutes", "mean"))
)

This reads as: group rows by channel, then calculate one mean for each group.

Polars

Polars is a DataFrame library built around expressions. It supports:

• eager execution: run each operation now;
• lazy execution: build a query plan, optimise it, then execute with collect().

import polars as pl

summary = (
    pl.scan_parquet("tickets.parquet")
    .filter(pl.col("minutes") >= 0)
    .group_by("channel")
    .agg(pl.col("minutes").mean().alias("mean_minutes"))
    .collect()
)

scan_parquet begins lazily. Polars can push the filter and selected columns towards the file read, avoiding unnecessary work.

Which tool?

This is not a winner-takes-all choice. Measure with representative data, include developer clarity, and avoid converting between libraries repeatedly.

Part 4 — Cleaning preserves meaning

Cleaning is a set of explicit transformations, not a command that makes data “good”.

A safe cleaning sequence:

1. Preserve immutable raw input and source metadata.
2. Validate the expected schema.
3. Parse types explicitly.
4. Standardise units and categorical spelling.
5. Identify duplicates with a documented identity rule.
6. Investigate missing and unusual values.
7. Produce a quality report.
8. Save the transformation code and resulting dataset version.

Duplicates are a definition problem

Exact duplicate rows may be repeated ingestion.

Two rows with the same person and timestamp may be retries. Two identical messages from different users may be legitimate.

Define:

• primary key: field or fields that identify one intended row;
• event identity: what counts as the same real-world event;
• deduplication window: whether repeated events within a time window count once.

Never call drop_duplicates() without explaining which columns define sameness and which row should survive.

Part 5 — Missing values carry information

A missing valueMissing valueA value that was not observed, recorded, available, or applicable. The reason it is missing matters; deleting or filling every missing value with one rule can hide bias or create misleading data. means no usable value was recorded. It may mean:

• the field was not applicable;
• a sensor failed;
• a user declined;
• an older system did not collect it;
• a join found no matching row;
• a parsing rule rejected the input.

Three common mechanisms are useful mental models:

• MCAR — missing completely at random: missingness is unrelated to observed or hidden values.
• MAR — missing at random given observed data: missingness relates to fields you observed.
• MNAR — missing not at random: missingness still relates to the missing value after considering observed fields.

These are assumptions, not labels a library can prove.

Possible actions:

• keep a native missing value;
• add a missingness indicator;
• impute with a training-set statistic;
• use a model that handles missing values;
• exclude a field or row with a documented reason;
• fix the upstream collection problem.

Do not replace every blank with zero. Zero may be a real measurement.

Part 6 — Outliers can be errors or important cases

An outlier is unusually far from the main data pattern under a chosen definition.

It might be:

• a unit error, such as seconds stored as milliseconds;
• a parsing error;
• fraud or abuse;
• a rare but valid customer;
• the exact failure case the system must handle.

Useful tools include quantiles, robust median-based summaries, scatter plots, domain limits, logarithmic transforms, and clipping.

Do not remove a row merely because it is more than three standard deviations away. First ask whether it is valid and whether production will contain similar cases.

If you cap, transform, or exclude values, fit thresholds on training data and record the rule.

Part 7 — Split before learning from the data

A training workflow usually separates:

• training set: fits feature and model parameters;
• validation set: chooses models, prompts, thresholds, and settings;
• test set: estimates final performance after choices are frozen.

The test set is not a recurring dashboard for model development.

Random, time, and group splits

A random split is suitable only when rows are sufficiently independent and future deployment resembles a random draw.

Use a time split when predicting future events:

train: Jan–Sep → validation: Oct → test: Nov

Use a group split when several rows belong to one person, patient, document, device, company, or conversation. Entire groups must stay together.

Sometimes both time and group boundaries matter.

What counts as “fit”?

Any operation that learns from data:

• mean or median imputation;
• scaling statistics;
• category vocabulary;
• target encoding;
• feature selection;
• text vocabulary or tokenizer training;
• dimensionality reduction;
• outlier thresholds;
• model parameters.

Hard-coded parsing rules can be applied consistently, but even those should be versioned and tested.

Part 8 — Leakage creates fake performance

Data leakageData leakageInformation reaches model training that would not be available when a real prediction is made, or a holdout set influences training decisions. Leakage produces unrealistically strong evaluation results that usually fail in production. occurs when training gains information unavailable at the real prediction moment, or when holdout data influences model choices.

Common forms:

1. Target leakage: a field directly or indirectly reveals the answer. refund_issued_at leaks whether a refund occurred.
2. Future leakage: a future event appears in historical features.
3. Preprocessing leakage: scaling, imputation, selection, or vocabulary is fitted before splitting.
4. Entity leakage: the same user or document family appears in training and test.
5. Duplicate leakage: exact or near duplicates cross the boundary.
6. Label leakage: annotator notes or file paths contain the class.
7. Evaluation leakage: test results repeatedly guide changes.

A practical leakage audit

For every feature, write:

available_at = timestamp when the value becomes knowable
prediction_at = timestamp when the model must decide

Reject features where available_at > prediction_at.

Also:

• sort columns by suspiciously strong relationship with the target;
• inspect feature names and lineage;
• train a simple baseline to expose impossible accuracy;
• check duplicate and nearest-neighbour overlap;
• split by time and group;
• compare offline feature availability with serving code.

Leakage detection needs domain knowledge. A generic checker cannot know when a business event becomes available.

Expected output:

A ['clean']
B ['clean']
C ['customer crosses split', 'feature comes from future']
D ['clean']

Part 9 — Feature engineering must be reproducible

Feature engineeringFeature engineeringTurning raw fields into useful model inputs using domain knowledge and reproducible transformations. Any learned transformation, such as a mean or vocabulary, must be fitted on training data only. turns raw fields into model inputs.

For a support ticket, raw fields might become:

• message character count;
• hour of day;
• days since account creation;
• known language;
• count of previous contacts before the prediction time;
• embedding of ticket text.

Good features are available at prediction time, stable enough to serve, documented, privacy-aware, and calculated identically in training and production.

Fit and transform

A useful interface separates:

pipeline.fit(training_rows)
pipeline.transform(any_rows)

fit learns state. transform reuses that state.

train_median = train["age"].median()

train["age_missing"] = train["age"].isna()
valid["age_missing"] = valid["age"].isna()

train["age"] = train["age"].fillna(train_median)
valid["age"] = valid["age"].fillna(train_median)

The validation median is never calculated. That keeps the validation set from teaching the pipeline.

In production, package the median, missing-indicator rule, code version, and expected input schema with the model.

Part 10 — EDA asks questions before modelling

Exploratory data analysisExploratory data analysisInspecting a dataset with summaries and visualisations to discover its shape, quality problems, patterns, and limitations before modelling. Holdout labels should remain protected from repeated exploration., shortened to EDA, uses summaries and visualisations to discover data shape and problems.

Begin with:

• row count, column count, types, and memory;
• unique-key violations;
• missingness by field, time, group, and source;
• category frequencies and unknown values;
• numeric range, quantiles, and unit checks;
• label balance;
• duplicate and near-duplicate rates;
• time coverage and collection changes;
• group coverage and representation.

Then inspect relationships with the target using training data.

Protect the holdout during EDA

Use the test set for final evaluation, not repeated plots and feature ideas.

If you inspect test labels, the knowledge changes your decisions. The test set has become part of development. Create a fresh holdout if possible and record what happened.

EDA can reveal association. It does not prove causation.

Worked example — prepare resume-match data

Suppose one row is a resume–job pair. The target is whether a qualified recruiter approved the match.

1. Define the prediction moment

The system scores the pair immediately after the job is published.

Interview outcome and later application status are unavailable, so they cannot be features.

2. Define identity

pair_id = hash(candidate_id, job_id, matching_policy_version)

Raw names and email addresses are not needed by the matching model.

3. Split safely

• keep all rows for one candidate in one split;
• train on older jobs;
• validate on a later period;
• test on the newest untouched period.

This tests new candidates and changing job language more honestly than a row-level random split.

4. Fit transformations on training

• learn skill vocabulary from training text;
• fit missing-experience handling from training rows;
• create text embeddings with one recorded model version;
• choose similarity thresholds using validation.

5. Audit

Check for:

• application-status or interview fields;
• duplicate resumes across splits;
• a source identifier that encodes recruiter decisions;
• groups absent from training;
• performance by relevant segments;
• personal text accidentally written to logs.

6. Freeze and evaluate

After choices are fixed, run once on test data. Store dataset version, feature code commit, split rules, contract version, model version, and evaluation report.

Part 11 — Version data, code, and lineage together

Data versioningData versioningRecording which exact dataset snapshot, transformation code, configuration, and lineage produced an experiment or release, so the result can be reproduced and audited. identifies the exact data used for an experiment.

Git is excellent for small text and code, but large datasets do not fit ordinary Git history well.

DVC-style artefact versioning

DVC, or Data Version Control, stores lightweight metadata in Git while large artefacts live in remote storage.

A reproducible stage can declare:

inputs + command + parameters → outputs + metrics

Checking out a Git commit and pulling matching artefacts recreates that version, assuming sources and environments remain available.

Delta-style table versioning

Delta Lake adds a transaction log to files in object storage. It supports consistent table updates, schema controls, and reading earlier table versions.

Use it when data is managed as a changing analytical table with concurrent readers and writers.

They solve related, different jobs

Neither automatically guarantees semantic correctness, privacy, or reproducibility. Also record:

• source snapshot and extraction query;
• transformation code commit;
• configuration and random seed;
• environment and library versions;
• schema and contract versions;
• checksums and lineage;
• access controls and retention.

Part 12 — Data contracts make assumptions executable

A data contractData contractA machine-checkable agreement about a dataset, including field names, types, null rules, allowed values, identity, freshness, and ownership. It turns unexpected upstream changes into visible failures instead of silent corruption. is a machine-checkable agreement between producers and consumers.

It can specify:

• field names and types;
• required, nullable, and defaulted fields;
• units and timezone;
• allowed categories and numeric ranges;
• primary key and uniqueness;
• freshness and expected volume;
• ownership and support contact;
• privacy classification;
• compatibility and version policy.

A contract failure should say what failed, where, how many rows are affected, and what safe action occurred.

Do not silently map every unknown category to a familiar one. UNKNOWN should mean unknown, not “the pipeline guessed”.

Part 13 — Schema evolution is a rollout

Schema evolutionSchema evolutionThe managed change of a dataset structure over time. Safe evolution considers old and new producers and consumers, versions breaking changes, and tests compatibility before rollout. is controlled change to a dataset structure.

Changes can be:

• additive: add nullable language;
• restrictive: make message non-null;
• representational: change created_at from local text to UTC timestamp;
• semantic: change what resolved means;
• destructive: rename or remove customer_id.

Adding a nullable field is often backward compatible for readers that ignore unknown columns. A semantic change may be breaking even if the type stays identical.

A safe rollout:

1. Propose the new meaning and owner.
2. Version and test the contract.
3. Make consumers tolerant where appropriate.
4. Deploy producers writing both forms if migration requires it.
5. Backfill and verify.
6. Migrate readers.
7. Monitor unknown, null, and failure rates.
8. Remove the old field after an agreed window.

Never reuse a field name for a different meaning.

Part 14 — Minimise and protect personal data

Personally identifiable informationPersonally identifiable informationData that identifies a person directly or can reasonably be combined to identify one, such as a name, email address, precise location, or stable account identifier. It should be minimised and protected throughout its lifecycle., shortened to PII, identifies a person directly or through combination.

Examples include names, email addresses, telephone numbers, precise locations, stable account IDs, and free text containing personal details.

Sensitivity depends on context and jurisdiction. Work with your organisation’s privacy and legal specialists.

Engineering principles:

• minimise: collect only fields needed for a stated purpose;
• separate: keep direct identifiers away from modelling data;
• pseudonymise: replace identifiers with controlled tokens where linkage is needed;
• encrypt: protect data in transit and at rest;
• restrict: grant least-privilege access and audit it;
• retain deliberately: define deletion and expiry;
• redact logs: do not copy raw records into errors, traces, or prompts;
• support rights and incidents: maintain lineage so records can be found and removed where required.

Anonymisation aims to prevent re-identification. Replacing a name with a stable hash is usually pseudonymisation, not guaranteed anonymisation: the same person remains linkable and source values may be guessable.

Deleting a direct name does not remove all privacy risk. Rare combinations can still identify people.

In a real AI system

Common mistakes

Exercises

Knowledge check

Practice activity

rows = [
    {
        "ticket_id": "t1",
        "customer": "u1",
        "email": "a@example.test",
        "message": "Cannot sign in",
        "prediction_day": 2,
        "resolved_day": 4,
        "split": "train",
    },
    {
        "ticket_id": "t1",
        "customer": "u2",
        "email": "b@example.test",
        "message": None,
        "prediction_day": 3,
        "resolved_day": 3,
        "split": "train",
    },
    {
        "ticket_id": "t3",
        "customer": "u1",
        "email": "a@example.test",
        "message": "Still locked out",
        "prediction_day": 5,
        "resolved_day": 6,
        "split": "test",
    },
]

# Add audit checks here.

from collections import Counter

ids = Counter(row["ticket_id"] for row in rows)
duplicate_ids = sorted(key for key, count in ids.items() if count > 1)

train_customers = {
    row["customer"] for row in rows if row["split"] == "train"
}
test_customers = {
    row["customer"] for row in rows if row["split"] == "test"
}

future_fields = [
    row["ticket_id"]
    for row in rows
    if row["resolved_day"] > row["prediction_day"]
]
missing_messages = sum(row["message"] is None for row in rows)

print("duplicate IDs:", duplicate_ids)
print("customers crossing splits:", sorted(train_customers & test_customers))
print("post-prediction resolution fields:", future_fields)
print("missing messages:", missing_messages)
print("direct identifier to remove/tokenise: email")

duplicate IDs: ['t1']
customers crossing splits: ['u1']
post-prediction resolution fields: ['t1', 't3']
missing messages: 1
direct identifier to remove/tokenise: email

Key takeaways

Flashcards

One-page sketchnote summary

train, valid, test = split(raw)
pipeline.fit(train)
X_train = pipeline.transform(train)
X_valid = pipeline.transform(valid)
X_test  = pipeline.transform(test)

Resources