Vision: Convolutions and Tokens

The same arithmetic on the house image, with three classic filters:

A convolutional layer runs many filters at once, and training chooses their weights: Part 4's loss discovers the edge detector from the data, because edge features lower the loss on later tasks.

Stacking the operation lets each layer detect larger patterns. The first layer reads pixels and finds edges; the next layer reads the first layer's outputs, so its filters see combinations of edges: corners and textures; deeper layers see parts, then whole objects. Zeiler and Fergus (2014) made the ladder visible by tracing each layer's strongest responses back to the pixels.

Between layers, pooling shrinks the map by keeping each small region's largest response: a 2 by 2 pooling over (320,\,0,\,320,\,0) keeps 320. Pooling discards the exact position, so the network gains tolerance to small shifts while the map shrinks to a quarter of its values.

The activation function from Part 5 sits after every layer, because stacked linear filters would collapse into one filter, the same collapse Part 5 proved on matrices.

David Hubel and Torsten Wiesel recorded from single neurons in a cat's visual cortex (1959) and found cells tuned to oriented edges at specific positions; their 1962 paper sorted them into simple and complex cells, a hierarchy from edges toward tolerance of position. They shared in the 1981 Nobel Prize in Physiology or Medicine (one half went to Roger Sperry, one quarter to each of them).

Kunihiko Fukushima built the hierarchy into a machine, the neocognitron (1980), alternating detecting layers and tolerance layers. Yann LeCun and colleagues added Part 4's backpropagation: their 1989 network read handwritten ZIP code digits, and LeNet-5 (LeCun, Bottou, Bengio, and Haffner, 1998) read, in the paper's words, "several million checks per day" in NCR's deployed systems.

The training set from that work, MNIST, holds 60,000 handwritten digits at 28 by 28 pixels each and remains the field's first classroom exercise. The 28 by 28 size returns in 2026, in the token counting of a frontier assistant.

AlexNet (Krizhevsky, Sutskever, and Hinton, 2012), which Part 6 showed training on two GTX 580s, stacked five convolutional layers and three fully connected layers: 60 million weights. The final layers are Part 3's classifier, a weighted sum and softmax over 1,000 object names, so every part of the machine is arithmetic from Parts 3 and 5.

1. Convolutional layers find edges, textures, and parts, with pooling between them.
2. The feature map flattens into one long vector, Part 5's list.
3. Fully connected layers and a softmax map the vector to 1,000 probabilities, exactly Part 3's classifier built from Part 5's matrices.

Training follows Part 6 step for step: cross entropy on labeled images, a backward sweep, a weight update, all on GPUs.

Everything in the pipeline is Part 5. A tiny worked case, a 2 by 2 patch with illustrative values:

The patch flattens into a list of 4 numbers, and the embedding matrix E, learned like every other weight, multiplies the list and produces a vector of the model's embedding width. With the illustrative E=\begin{pmatrix}1&0&0&1\\0&0.1&0.1&0\end{pmatrix} the patch becomes the vector (100,\,10): table times list, exactly as Part 5 taught.

A position embedding is added so the model knows which patch sat where, exactly as word tokens carry positions. From this point on, the transformer sees no difference between a patch token and a word token.

A picture enters the model as 196 vectors, and the machinery that reads them is the gray box Part 9 opens.

Convolution carries built in assumptions: nearby pixels matter together, and a pattern means the same thing anywhere. The Vision Transformer discards both and lets attention learn where to look, so it must relearn locality from data.

The paper measured the cost in training data and the gain in accuracy. Trained on ImageNet alone, the transformer trails comparable convolutional networks, because transformers "do not generalize well when trained on insufficient amounts of data"; pretrained on 14 million to 300 million images, it matches or beats them, and the authors conclude that "large scale training trumps inductive bias."

When students photograph their homework, the assistant reads the image as patch tokens. CLIP (Radford et al., announced January 5, 2021) trained a vision encoder by predicting which caption goes with which image across 400 million pairs; LLaVA (Liu et al., 2023) projected such patch features straight into a language model's embedding space, so image patches arrive as tokens among the words. The frontier assistants followed: GPT-4 announced image input on March 14, 2023, Gemini launched as natively multimodal on December 6, 2023, and Claude 3 shipped vision on March 4, 2024.

Part 7's context applies to pictures directly, because a picture is context. Anthropic's current documentation counts one visual token per 28 by 28 pixel patch, up to 1,568 tokens per image at the standard tier and 4,784 at the high resolution tier (2026); MNIST's whole digit, 28 by 28, is exactly one such patch.

OpenAI's documentation counts 32 by 32 pixel patches for its smaller models and 512 by 512 tiles for the larger ones; the patch sizes differ, while the billing idea is the same: pictures are counted in patches.