Move with the arrow keys, a presentation clicker, or the buttons at the bottom left; the gear at the bottom right opens the slide settings.
Today we open the recipe: what a mistake costs, and how every weight receives its nudge.
Part 3 gave mistakes a magnitude: the model missed the right answer by 1.
Both halves of that question have real answers, and both are choices we make; the function that scores a mistake comes first.
The recorded correct answer is called the ground truth. A loss function is a function that reads the ground truth and the prediction and returns one number, the loss: zero for a perfect answer, larger for a worse one. Training has exactly one goal: make the average loss over the training data small.
| ground truth | prediction | mistake | loss = magnitude of the mistake |
|---|---|---|---|
| 67 | 66 | −1 | 1 |
| 67 | 67 | 0 | 0 |
| 67 | 69 | 2 | 2 |
| 67 | 80 | 13 | 13 |
This first loss scores a mistake by its magnitude alone; the next slide meets its main rival.
Two natural choices score a mistake of magnitude m: the absolute error, m itself, and the squared error, m times m. The plot shows the price of each vote: under the square, a mistake of 2 costs 4, and a mistake of 5 costs 25, so large mistakes dominate the training.
The square has a second gift, visible at the bottom of the plot: its steepness shrinks as the mistake shrinks, so a learner slows down as it closes in; the V stays equally steep to the last instant and invites overshooting. The walk downhill uses this shrinking slope.
Both shapes are symmetric: a mistake of 2 over and a mistake of 2 under cost the same. Symmetry is a statement about the world, chosen on purpose: for 34 plus 33, answering 65 and answering 69 are equally wrong, so the loss should not care about direction.
A symmetric loss tells the learner: I have no preference between overshooting and undershooting; land close, from either side.
When the two directions cost differently in the world, the loss should tilt the same way. A cafeteria orders lunches before knowing the demand: a missing meal costs 1 (a student unfed), a wasted meal costs 0.2 (food thrown away).
The spam filter from Part 3 faces the same tilt: deleting a real message costs far more than passing one spam, so its training weights the two mistakes differently. Choosing a loss is deciding which mistakes you can live with.
The squared error is two centuries old, and an asteroid made it famous. Ceres, discovered by Giuseppe Piazzi on the first night of 1801, vanished into the Sun's glare after only weeks of observations. Carl Friedrich Gauss fitted an orbit to the scraps by making the squared errors small, and in December of that year Ceres reappeared close to his prediction.
Adrien Marie Legendre published the method first, in 1805, naming it least squares; Gauss then claimed he had used it since 1795, and the two argued about who initially discovered the idea for years.
The choice of the square was already a policy, in the sense that an orbit should treat overshooting and undershooting a star's position as equally bad.
Give the loss a landscape. A toy model predicts with one weight: \text{prediction} = w \cdot 10, ground truth 67, squared loss. Plotting loss against w draws a valley with its bottom at w = 6.7. Training is a walk: stand at w, measure the slope under your feet, and step downhill, the step proportional to the slope. In physics words: your velocity is set by the steepness; fast high on the wall, slower as the floor flattens, settled where the slope is zero. This walk is called gradient descent; gradient is the field's word for the slope.
| step | w | pred | loss | slope | new w |
|---|---|---|---|---|---|
| 1 | 3.00 | 30.0 | 1369 | −740 | 3.74 |
| 2 | 3.74 | 37.4 | 876 | −592 | 4.33 |
| 3 | 4.33 | 43.3 | 561 | −474 | 4.81 |
| 4 | 4.81 | 48.1 | 359 | −379 | 5.18 |
| … approaching w = 6.7, where the slope is 0 | |||||
Step size r = 0.001. The slope needs no calculus: nudge w a little, watch the loss, and divide the change by the nudge; calculus gives the exact value, the nudge gives a measurement anyone can take.
The step size, called the learning rate, decides the walk's character. For this valley the arithmetic is exact: each step multiplies the distance to the bottom by (1 - 200r). Three settings tell the whole story.
Plain descent resets its speed at every step: velocity from slope, with no "memory". Give the walker mass instead, and the physics changes by one line: the slope now sets the acceleration, and a little friction drains the speed.
The ball builds velocity down the long wall, coasts across dips that strand a walker, and settles because friction wins once the floor levels; practitioners call this momentum, in use since the 1980s.
| physics | training |
|---|---|
| position | the weight's value |
| height | the loss |
| steepness | the slope of the loss |
| velocity | the running step, remembered |
| acceleration | the slope's push on that step |
| friction | damping, so the ball settles |
With two weights the valley becomes a bowl, and the slope becomes an arrow pointing uphill in the weight plane; descent steps against the arrow. With millions of weights nothing changes except the bookkeeping: every step needs the slope of the loss with respect to every single weight.
A two stage toy: input 2, times w1 = 3 gives h = 6, times w2 = 4 gives the prediction 24; ground truth 20, so the mistake is 4 and the squared loss is 16. On a bicycle, if the pedals turn the wheel at ratio 2 and the wheel moves the road at ratio 4, the pedals move the road at ratio 8: ratios along a chain multiply. The same chain runs through the toy: w1 moves h at ratio 2 (the input), h moves the prediction at ratio 4 (that is w2), and the prediction moves the loss at ratio 8 (twice the mistake); so w1 moves the loss at 2 \cdot 4 \cdot 8 = 64.
| question | nudge test (nudge 0.01) | multiplied ratios |
|---|---|---|
| prediction per w1 | 0.08 / 0.01 = 8 | 2 × 4 = 8 |
| loss per w2 | about 48.4 | 6 × 8 = 48 |
| loss per w1 | about 64.6 | 2 × 4 × 8 = 64 |
The nudge test lands near the product, and closer as the nudge shrinks; calculus gives the product exactly.
In a deep network, every weight's ratio product shares the same tail: the ratios from the loss back to that weight's layer. So compute once, from the loss backward, layer by layer, reusing the shared products as you go; when the sweep reaches the input, every weight holds its slope. The procedure is called backpropagation, short for backward propagation of errors.
The count is the whole point: the nudge test needs one forward pass per weight, a billion passes for a billion weights; backpropagation delivers every slope for about the cost of two passes, one forward and one backward.
The 1986 paper by Rumelhart, Hinton, and Williams made the method standard; the idea had appeared earlier, in Paul Werbos's 1974 thesis.
Part 2 promised: the correction signal weakens as it travels backward through many steps. Now the arithmetic is yours: the backward sweep multiplies one local ratio per step, so long products of small ratios shrink toward nothing while long products of large ratios explode.
"Deeper is always better." Because every extra layer multiplies one more ratio into the backward product, careless depth starves the early layers of signal or blows the nudges up; the table shows both fates.
| steps traveled | each ratio 0.5: product | each ratio 2: product |
|---|---|---|
| 1 | 0.5 | 2 |
| 2 | 0.25 | 4 |
| 3 | 0.125 | 8 |
| 5 | 0.03 | 32 |
| 10 | 0.001 | 1024 |
The first column is the vanishing gradient from Part 2: the earliest weights receive almost no nudge; the second is its twin, the exploding gradient. The gated designs from Part 2, among them the LSTM, answer exactly this product; so do careful learning rates.
Training prints one plot above all others: loss against training steps. You can now read it like terrain. The early cliff is the walk down the high wall; the long tail is the flat valley floor; a sudden spike is a step that jumped too far and landed on a wall.
Which loss trains the language model itself? The game from Part 1 scores like this: after the true next token is revealed, the loss depends on the probability the model had given it: large probability, small loss; small probability, enormous loss.
| probability given to the true token | loss |
|---|---|
| 1.00 | 0 |
| 0.90 | 0.11 |
| 0.50 | 0.69 |
| 0.10 | 2.3 |
| 0.01 | 4.6 |
Computed with the logarithm formula the field uses; its details arrive in Part 5.
Notice the tilt: this loss has no symmetric partner, because there is no such thing as overshooting a probability of 1. Being confidently wrong at 0.01 costs about forty times more than being nearly sure at 0.9. The principle holds at the top of the field: the chosen shape encodes chosen values, and the LLM's loss says that confident wrongness is the cardinal sin.
1. Under the squared loss with ground truth 67, what do predictions 65 and 69 cost, and what does the equality tell you about the loss?
4 and 4; the loss is symmetric, so direction carries no cost.
2. The cafeteria again: shortage costs 1 per meal, waste costs 0.2, and demand is 100 or 120 with equal chances. Compute both expected costs and choose the stock.
Stocking 100 carries an expected cost of 10; stocking 120 carries 2; order 120, because the tilt pushes the best answer up.
3. A network has one million weights. How many forward passes does one training step cost with the nudge test, and about how many passes with backpropagation?
About a million and one against about two.
Part 5 slows training down to its arithmetic: vectors, matrices, softmax, and the backward sweep at full width. Attention keeps its appointment in Part 9.
Slides: Mehmet Kerem Turkcan, Center for Smart Streetscapes, Columbia University; keremturkcan.com; mkt2126@columbia.edu