CNNs and computer vision basics

A 32 × 32 colour image contains 3,072 numbers. A fully connected layer can read all of them, but it begins with no built-in reason to treat neighbouring pixels as related or to reuse an edge detector in the top-left and bottom-right corners.

A convolutional neural network starts with those two useful assumptions: nearby values form local patterns, and the same local pattern may matter at many positions.

Why this matters

Computer visionComputer visionThe area of computing concerned with extracting useful information from images or video. A vision model still needs a clearly defined task, representative data, and measured errors. is the area of computing concerned with extracting useful information from images or video. Classification assigns a label to an image. Detection locates objects. Segmentation assigns labels to pixels or regions. Other tasks estimate depth, motion, keypoints, similarity, or generated content.

The model does not receive “a cat” or “a cracked component.” It receives an array of measured values. A useful vision system must connect those values to a precisely defined target, then survive changes in cameras, lighting, position, background, and deployment policy.

Plain-English mental model

Step 1 — An image is a tensor with spatial structure

A greyscale image can be represented as one grid of intensities. A colour image often has three channels: red, green, and blue. A channel is one aligned grid of measurements.

PyTorch convolution layers normally use channel-first order:

one image:  (channels, height, width)
one batch:  (batch, channels, height, width)

A batch of sixteen 32 × 32 RGB images has shape (16, 3, 32, 32). The first 3 is not the number of examples; it is the number of aligned input channels.

Pixel values also need a declared scale and preprocessing contract. A pretrained model may expect a particular resize, crop, value range, and per-channel normalisation. Correct tensor shape does not compensate for incorrect preprocessing.

Image coordinates and labels deserve the same care. If an augmentation flips or crops the image, any bounding boxes, masks, or keypoints must follow the same geometry.

Step 2 — Slide, multiply, and sum

A convolutional kernelConvolutional kernelA small learned grid of weights, also called a filter, that is reused at every spatial position to detect the same kind of local pattern., also called a filter, is a small grid of weights. At one position:

1. Take the local input patch under the filter.
2. Multiply corresponding patch values and filter weights.
3. Add all the products.
4. Optionally add one learned bias.

The result is one output value. Move the same filter and repeat.

Deep-learning libraries commonly perform cross-correlation: they use the learned filter in its stored orientation rather than mathematically flipping it first. The operation is still conventionally called convolution. Because the weights are learned, this naming difference does not change the basic learning story, but it matters when reproducing a hand calculation.

A feature mapFeature mapOne spatial output channel produced by applying a learned filter across an input. Its positions record where that filter responded and by how much. is one spatial output channel. Its positions record where one learned filter responded and by how much. Eight output filters create eight feature maps.

Worked example — detect a vertical boundary

Use this 5 × 5 input, where 0 changes to 1 across a vertical boundary:

0 0 1 1 1
0 0 1 1 1
0 0 1 1 1
0 0 1 1 1
0 0 1 1 1

Slide this 3 × 3 vertical-edge filter with stride 1 and no padding:

-1  0  1
-1  0  1
-1  0  1

At the top-left position, each row contributes 0×−1 + 0×0 + 1×1 = 1. Three rows sum to 3. Once the filter lies entirely inside a constant region of ones, its negative and positive sides cancel to 0.

The hand-written weights make the example interpretable. In an ordinary CNN, backpropagation learns the filter values that reduce the training loss.

Now inspect every product interactively. Change to a horizontal input while keeping the vertical-edge filter, then change only the filter. The matching orientation should respond differently. Select any output cell to expose its patch arithmetic.

Step 3 — Stride and padding control the map size

StrideStrideThe number of input positions a convolution or pooling window moves between neighbouring outputs. A larger stride usually creates a smaller spatial output. is how far the filter moves between neighbouring outputs. Stride 1 visits every position. Stride 2 skips alternate positions and usually makes a smaller output.

PaddingPaddingExtra values placed around an input before a local operation. Zero padding can preserve more border positions, but the added zeros are not observed pixels. adds values around the input before the local operation. Zero padding adds an explicit border of zeros. It lets a filter be centred near the original edge and can preserve spatial size for suitable settings.

For one spatial axis, dilation 1, and a square or rectangular operation applied independently to height and width:

output = floor((input + 2 × padding − kernel) ÷ stride) + 1

The floor matters when the filter does not land evenly at the final position.

def output_size(input_size, kernel, stride=1, padding=0):
    return (input_size + 2 * padding - kernel) // stride + 1

settings = [
    ('valid', 5, 3, 1, 0),
    ('padded', 5, 3, 1, 1),
    ('strided', 5, 3, 2, 1),
    ('image conv', 32, 3, 1, 1),
    ('then pool', 32, 2, 2, 0),
]

for name, size, kernel, stride, padding in settings:
    result = output_size(size, kernel, stride, padding)
    print(f'{name}: {size} -> {result}')

Padding is a numeric boundary assumption. A zero outside the image is not a measured black pixel, empty road, healthy tissue, or absence of an object. Border-sensitive tasks may need careful padding and explicit error analysis near image edges.

Dilation spaces filter taps apart, allowing a filter to cover a wider region without adding weights. It belongs in the full output-size formula, but this beginner calculation keeps dilation at its default value of 1.

Step 4 — Channels, filters, and parameter sharing

For RGB input, one ordinary filter has a separate 2D weight grid for each of the three input channels. Those three grids are multiplied with the aligned local RGB patch and summed into one output position.

PyTorch stores ordinary Conv2d weights with shape:

(output channels, input channels, kernel height, kernel width)

A layer with 3 input channels, 8 output channels, 3 × 3 kernels, and a bias has:

weights = 8 × 3 × 3 × 3 = 216
biases  = 8
total   = 224 learned parameters

input_channels = 3
output_channels = 8
kernel_height = 3
kernel_width = 3
uses_bias = True

per_filter = input_channels * kernel_height * kernel_width
total = output_channels * per_filter
if uses_bias:
    total += output_channels

print('weight shape:', (output_channels, input_channels, kernel_height, kernel_width))
print('learned parameters:', total)

The parameter count does not depend on image width or height because the same weights are reused. A larger image creates more arithmetic and a larger feature map, but not more filter parameters.

This parameter sharing gives a convolution a useful position-related behaviour: if a pattern moves, the corresponding response often moves too. Padding, stride, pooling, borders, and later global aggregation complicate that relationship. A CNN is not automatically invariant to every translation, rotation, scale, or viewpoint.

Step 5 — Pooling summarises local regions

PoolingPoolingA fixed local summary operation that reduces spatial resolution, such as keeping the maximum or average value in each window. Pooling has no learned filter weights. is a fixed local summary operation. It has no learned filter weights.

Max pooling keeps the largest value in each window. Average pooling keeps the mean. With a 2 × 2 window and stride 2, this 4 × 4 map becomes 2 × 2:

feature_map = [
    [1, 3, 2, 0],
    [4, 6, 1, 2],
    [0, 2, 5, 3],
    [1, 1, 2, 4],
]

def pool(values, reducer):
    output = []
    for top in range(0, len(values), 2):
        row = []
        for left in range(0, len(values[0]), 2):
            window = [
                values[top + dy][left + dx]
                for dy in range(2)
                for dx in range(2)
            ]
            row.append(reducer(window))
        output.append(row)
    return output

maximum = pool(feature_map, max)
average = pool(feature_map, lambda values: sum(values) / len(values))

print('max:', maximum)
print('average:', average)

Pooling reduces spatial resolution and discards detail. That can reduce computation and make a later prediction less sensitive to tiny movements, but it can harm tasks that need exact boundaries or small objects. Strided convolutions are another learned way to reduce resolution.

Adaptive average pooling requests an output size rather than a fixed window. AdaptiveAvgPool2d((1, 1)) reduces each channel to one number, allowing a classifier head to accept more than one input height and width. It still discards spatial layout.

Step 6 — Deeper layers can use wider context

The receptive fieldReceptive fieldThe region of the original input that can influence a particular activation. Stacking local layers and using stride can increase this theoretical region. of an activation is the region of the original input that can influence it. One 3 × 3 convolution sees a local 3 × 3 patch. A later 3 × 3 convolution reads earlier activations that already summarise neighbouring patches, so its input influence is wider.

Track two quantities:

• Receptive field: how many original input positions can influence the current activation.
• Jump: how far apart neighbouring current activations are in original input coordinates.

For each layer with kernel k and stride s:

new receptive field = old receptive field + (k − 1) × old jump
new jump            = old jump × s

layers = [
    ('conv 3x3', 3, 1),
    ('pool 2x2', 2, 2),
    ('conv 3x3', 3, 1),
]

receptive_field = 1
jump = 1

for name, kernel, stride in layers:
    receptive_field += (kernel - 1) * jump
    jump *= stride
    print(f'{name}: receptive field {receptive_field}, jump {jump}')

A wide theoretical receptive field does not prove the model used the right evidence. Occlusion tests, error slices, saliency methods used cautiously, and targeted counterexamples can help investigate reliance, but none provides a complete causal explanation by itself.

Step 7 — Residual connections preserve an identity route

A residual connectionResidual connectionA shortcut that adds a block's input to a learned transformation of that input, producing x plus F(x). The two paths need compatible shapes for addition., also called a skip connection, adds a block’s input to the block’s learned transformation:

output = x + F(x)

Instead of requiring the learned branch to recreate an entire useful representation, it can learn a change F(x) while the original x has a direct route. This supports optimisation of deeper networks; it does not guarantee that an arbitrarily deep model will generalise.

Addition requires compatible shapes. If channel count or spatial size changes, the shortcut must be transformed deliberately—often with a learned projection—or the values cannot be added element by element.

The arithmetic is ordinary:

x = [1.0, -2.0, 0.5]
residual = [0.2, 0.5, -0.1]
output = [original + change for original, change in zip(x, residual)]

print(output)

Expected output:

[1.2, -1.5, 0.4]

In a real residual block, F is a learned sequence such as normalisation, activation, and convolutions. Exact ordering varies by architecture.

Step 8 — Build a shape-readable CNN in PyTorch

This small classifier preserves 32 × 32 size in each convolution, halves it once with max pooling, then uses adaptive average pooling to produce one value per final channel.

import torch
from torch import nn

class SmallCNN(nn.Module):
    def __init__(self, classes: int):
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 8, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2),
            nn.Conv2d(8, 16, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.AdaptiveAvgPool2d((1, 1)),
        )
        self.classifier = nn.Linear(16, classes)

    def forward(self, images):
        maps = self.features(images)
        features = torch.flatten(maps, start_dim=1)
        return self.classifier(features)

model = SmallCNN(classes=4)
images = torch.zeros(2, 3, 32, 32)

with torch.no_grad():
    feature_maps = model.features(images)
    logits = model(images)

print('feature maps:', tuple(feature_maps.shape))
print('logits:', tuple(logits.shape))

logits contains four unnormalised class scores per image. It is not a probability and does not include the true label. During training, a classification loss compares those scores with targets. During evaluation, restore the selected checkpoint, call model.eval(), disable gradient tracking, and report protected measures beyond aggregate accuracy when the task requires them.

Assert shapes at important boundaries. A silent height/width or channel swap can produce plausible-looking numbers while teaching the wrong contract.

Step 9 — Transfer learning reuses a visual starting point

Transfer learningTransfer learningStarting a new task from parameters learned on an earlier task instead of starting every parameter randomly. The source data and preprocessing may not match the new deployment setting. starts a new task from parameters learned on an earlier task. A pretrained backbone may already contain useful local and compositional visual features.

Two common baselines are:

1. Fixed feature extractor: freeze the pretrained backbone and train only a new task-specific head.
2. Fine-tuningFine-tuningContinuing training of some or all pretrained parameters on a new task, usually with a controlled learning rate and task-specific validation evidence.: continue training some or all pretrained parameters on the new task.

Start with a named weights version. Torchvision weight objects bundle the preprocessing associated with that version. DEFAULT can point to a different weights version in a future library release, so record the resolved enum and package versions for reproducibility.

from torch import nn
from torch.optim import AdamW
from torchvision.models import resnet18, ResNet18_Weights

weights = ResNet18_Weights.IMAGENET1K_V1
preprocess = weights.transforms()
model = resnet18(weights=weights)

for parameter in model.parameters():
    parameter.requires_grad = False

input_features = model.fc.in_features
model.fc = nn.Linear(input_features, 4)

optimizer = AdamW(model.fc.parameters(), lr=1e-3, weight_decay=1e-2)

trainable = [name for name, parameter in model.named_parameters()
             if parameter.requires_grad]
print('trainable tensors:', trainable)
print('output classes:', model.fc.out_features)

requires_grad = False freezes parameter gradients. It does not freeze mutable buffers such as BatchNorm running statistics. For a truly fixed feature extractor, keep the frozen backbone in evaluation mode while training the new head. A generic loop that calls model.train() on the entire object can otherwise keep updating those running statistics.

If fine-tuning later layers, declare exactly which blocks are unfrozen. A smaller learning rate for pretrained parameters is a hypothesis to validate, not a universal constant. Compare the fixed baseline and fine-tuned run under the same split, transform contract, checkpoint rule, and error slices.

Pretraining is not neutral. Review source labels, likely data overlap, licences, demographic and geographic coverage, imaging equipment, and harmful shortcuts. Reused features can transfer useful structure and unwanted bias.

Step 10 — Evaluate the vision system, not only the network

An image classifier’s score can collapse when deployment changes the camera, crop, compression, lighting, background, resolution, or class prevalence. Protect the entire input and decision contract:

• version image decoding, channel order, colour space, resize, crop, and normalisation;
• split by the real unit of generalisation, such as patient, device, site, video, product batch, or time—not random near-duplicate frames;
• inspect corrupted files, duplicates, label ambiguity, and metadata leakage;
• measure important classes, sites, devices, lighting conditions, object sizes, and border positions;
• calibrate thresholds and abstention policy for the downstream cost;
• monitor input drift, failure slices, latency, and human-review load after release.

Image models can learn backgrounds, rulers, scanner marks, watermarks, text overlays, or capture workflows instead of the intended object. Counterexamples that preserve the target while changing these shortcuts are powerful tests.

Real-world AI engineering example

Common mistakes

Knowledge check

Practice activity

Exercises

Key takeaways

Flashcards

One-page sketchnote summary

patch × shared filter → sum
maps → pool/stride → wider context
x + F(x) → classifier

Resources