Move with the arrow keys, a presentation clicker, or the buttons at the bottom left; the gear at the bottom right opens the slide settings.
Today we open the machine at its two ends: the way in for tokens, and the way out for answers.
Freeze the question and allow only two answers, and the giant game collapses: a vocabulary of 50,257 tokens becomes a vocabulary of 2.
A machine that assigns probabilities to a fixed set of answers is called a classifier.
For decades, spam filtering, loan screening, and medical screening ran on machines of exactly this kind, each task a game with two answers.
Our tiny machine will answer: is this message spam? It reads the message through three smaller questions, and it stores each answer as a number, 1 for yes and 0 for no.
| question about the message | answer | as a number |
|---|---|---|
| does it contain the word free? | yes | 1 |
| does it shout with !!! anywhere? | yes | 1 |
| is the sender in your contacts? | no | 0 |
Numbers that answer questions about the input are called features. Hold onto this move; it returns at a smaller scale before the lecture ends.
Each question carries a weight, a number that says how strongly its answer should push, and in which direction. One extra number, the bias, sets where the machine stands before it reads anything.
The weights here are illustrative; a real filter learns its own, as the coming slides show.
A score can be any number, while a probability must live between 0 and 1. The bridge is a fixed curve, called the sigmoid: feed it any score, and it returns a probability. Large positive scores land near 1; large negative scores land near 0; a score of 0 lands exactly on one half.
So message A reports spam 0.95, leaving 0.05 for the other answer; message B reports spam 0.03.
Because the signal flows from the features, through one weighted sum, to the answer, in a single pass with no loop, the design is called a feedforward network.
Training reuses the recipe from Part 2: show the machine a message with a known answer; when it says spam 0.95 for a real message from your friend, the mistake is large, so every weight gets nudged in the direction that shrinks it. After enough messages, the weights settle into values like ours.
Stacking several weighted sums and activation functions in a row, each feeding the next, lets the machine answer subtler questions than any single line can; the stack is still feedforward, and still one pass.
The machine on the last four slides is the original, give or take one detail: Rosenblatt's perceptron answered through a hard yes or no threshold, where ours reads a probability off the curve. In 1958 Frank Rosenblatt, at the Cornell Aeronautical Laboratory, published the perceptron and demonstrated it in simulation on an IBM 704; by 1960 his team had built it as hardware, the Mark I, whose weights were dials turned by electric motors. The press expected miracles.
In 1969, Marvin Minsky and Seymour Papert proved that a single weighted sum can never answer some simple questions, and funding for the whole direction collapsed for a decade. The repair was already known: stack layers. In 1986, David Rumelhart, Geoffrey Hinton, and Ronald Williams published the nudging recipe that trains a whole stack at once; your spam filter and the largest models of 2026 are trained with descendants of that recipe.
Under the hood nothing changed: 67 is one more token in the vocabulary, and the model scored it against every other token, exactly as it scored yes against no.
"Track the probability of the correct answer." Both models score 0.30, yet model A stays within 2 of the truth, while model B is usually absurd; because a scorecard over pure symbols carries no notion of nearness, almost right does not exist for it. The counted chains of Part 1 lived entirely in this symbol world.
Attach to each number token the value it names: the token 67 carries the number 67. Nearness finally exists: 68 sits at distance 1 from the truth, while 341 sits at distance 274.
The two models from the last slide can now be told apart with one number each: on average, model A misses the truth by about 0.9, while model B misses it by about 84.
A machine judged by the distance of its number from the truth is doing regression, and the nudging recipe works unchanged, because distance supplies the measured mistake it needs.
Ask one more question about each token: how many digits does it have? The token 67 now carries two numbers, (67, 2): its value, and its digit count. Two numbers place each token as a point in a plane, and every further question would add an axis.
Neighborhoods now mean something: 67 and 68 sit close on both axes, while 8 and 341 disagree on both.
The list of numbers a model attaches to a token is called the token's embedding. Ours has two slots, with questions we wrote ourselves: value, and digit count. A real model attaches a longer list: the 2019 language model whose tokenizer we met in Part 1, in its smallest version, gives every one of its 50,257 tokens a list of 768 numbers.
| token | slot 1 | slot 2 | … | slot 768 |
|---|---|---|---|---|
| cat | 0.62 | −1.13 | … | 0.28 |
| kitten | 0.58 | −1.09 | … | 0.31 |
| 67 | 1.02 | 0.44 | … | −0.77 |
| 68 | 1.05 | 0.47 | … | −0.74 |
Numbers illustrative; the pattern is the point: similar tokens carry similar rows, and reading a row is a table lookup.
In a real model nobody writes the questions. Because the slots are weights, gradient descent tunes them like every other weight, and the model discovers whichever questions help it predict the next token. Tokens that behave alike end up with similar lists, so they land near each other: cat near kitten, 67 near 68.
In 2013, Tomas Mikolov and colleagues at Google trained token embeddings from raw text and then did arithmetic on the lists themselves. Starting from king, subtracting man, and adding woman lands nearest to queen; starting from Paris, subtracting France, and adding Italy lands nearest to Rome. Nobody programmed those facts; the learned slots turned out to include questions about royalty, gender, and country.
Read the whole machine from Part 2 again, with today's parts in place. Tokens enter and become rows of the embedding table. The middle, which reads the whole window through attention, stays a gray box until Part 9. At the far end a feedforward readout, built from the same weighted sums as your spam filter, gives every token a score, and the softmax, the sigmoid's sibling for many answers, maps the scores to the familiar bars.
1. A filter scores a message at exactly 0. What probability does it report, and what does the bias contribute before any feature speaks?
One half; the bias sets where the machine stands before it reads anything, so it is the filter's starting lean.
2. Two models both give the correct token a probability of 0.30. What extra structure lets you prefer one, and which mapping created it?
Distance between answers, created by mapping each number token to the value it names.
3. In a real embedding, who writes the questions the slots answer?
Nobody; because the slots are weights, gradient descent tunes them until useful questions emerge.
Part 4 measures the mistake and traces it backward: loss functions and backpropagation.
Slides: Mehmet Kerem Turkcan, Center for Smart Streetscapes, Columbia University; keremturkcan.com; mkt2126@columbia.edu