Move through the deck with (i) the arrow keys, (ii) a presentation clicker, or (iii) the buttons at the bottom left. The gear at the bottom right opens the slide settings.
Position (2,1) means two meters east and one meter up. Velocity (3,4) means that position gains three meters east and four meters up each second.
A vector is a list of numbers. A list with two slots also describes an arrow on the page.
| (3,4)+(0,2)=(3,6) | add matching slots |
| 2(3,4)=(6,8) | scale every slot |
Part 2 called the RNN memory a list of numbers of fixed length. That memory is a vector.
History: Josiah Willard Gibbs privately printed Elements of Vector Analysis in two parts, from 1881 to 1884, for his Yale classes.
A ball starts at (0,2) with velocity (3,4). Gravity changes the velocity by (0,-10) each second, and one tick lasts 0.2 seconds.
| tick | position | velocity |
|---|---|---|
| 0 | (0, 2) | (3, 4) |
| 1 | (0.6, 2.8) | (3, 2) |
| 2 | (1.2, 3.2) | (3, 0) |
| 3 | (1.8, 3.2) | (3, −2) |
| 4 | (2.4, 2.8) | (3, −4) |
| 5 | (3.0, 2.0) | (3, −6) |
At each tick, update the position by adding velocity times the tick length. After the position update, update the velocity by adding gravity times the tick length. During training, the learning rate sets how far the weight vector moves along the update direction. The backward sweep computes the rates used to determine that direction.
A pedal spins at 3 turns per second, and a hand crank spins at 1 turn per second. Wheel A spins at twice the pedal speed. Wheel B spins at the pedal speed plus three times the crank speed.
A matrix is a table with one row per output and one column per input. For each row, multiply every entry by the matching input entry. The sum of those products is one output.
History: James Joseph Sylvester coined the mathematical word matrix in 1850.
Feed each column of box 1, the matrix on the right, through box 2, the matrix on the left. Those two answers become the columns of the combined table.
When the number of outputs from box 1 equals the number of inputs to box 2, multiplying the matrices produces another matrix.
History: Arthur Cayley defined matrix multiplication in his 1858 Memoir on the Theory of Matrices so that a product represents one transformation after another.
| product | shapes | calculation | illustrative example |
|---|---|---|---|
| row times list | row, list | Add the products of matching slots. | (2,1)(3,1)=7 |
| table times list | table, list | Each row times the input list produces one output. | the gearbox |
| elementwise product | list, list | Each pair of matching slots produces one output slot. | (5,2,4)\odot(1,0,1)=(5,0,4) |
The elementwise product multiplies each pair of matching slots independently. Because multiplication by zero gives zero, a zero in the mask blocks that slot's contribution during the backward sweep.
Every output doubles, because every collected contribution doubles.
Part 3 computed this illustrative spam score:
The weights form one row. The features form one list. The superscript \mathsf T marks the horizontal list as transposed, so the matrix reads it as a column. The bias is added after the row combines its inputs.
One neuron is one row.
A layer of neurons is a matrix with a bias vector.
| row | calculation | output |
|---|---|---|
| 1 | (2.1,1.4,-3.0)(1,1,0)^{\mathsf T}-0.5 | 3.0 |
| 2 | (0.5,-1.0,2.0)(1,1,0)^{\mathsf T}+1.0 | 0.5 |
The second row and its bias are illustrative. The matrix with two rows returns (3.0,0.5).
During an embedding lookup, the model fetches one row from the embedding table. GPT-2's smallest model, published in 2019, uses 50,257 token rows with 768 numbers in each row.
Call the daily table T. Columns describe today, and rows describe tomorrow. The sun column is (0.8,0.2), and the rain column is (0.4,0.6). The list (1,0) means that today is certainly sunny.
Use tomorrow's probability vector as the next input:
T^2 gives the chances of sun and rain two days later.
Where paths merge, their contributions add: 0.8(0.8)+0.4(0.2)=0.72.
History: Andrey Markov studied chains of dependent events in 1906. Sergei Bernstein used the name Markov chain in 1926.
Reading note: some textbooks place today's state in each row. Fix the convention before comparing tables.
“Deeper always means smarter.”
Suppose two layers only collect weighted sums. Could one table replace both?
For every input, the stack and its product matrix produce the same output. Stacked linear layers therefore reduce to one layer.
| A rectifier between the tables | |
|---|---|
| x=(0,1) | (1,-1)\to(1,0)\to(1,1) |
| -x=(0,-1) | (-1,1)\to(0,1)\to(0,1) |
Here the stack sends x=(0,1) to (1,1) and -x to (0,1). Because any matrix that sends x to (1,1) must send -x to (-1,-1), no single matrix matches both inputs.
Once a nonlinear activation function separates the tables, the stack can represent additional functions.
History: Glorot, Bordes, and Bengio published rectifier network results in 2011. Krizhevsky, Sutskever, and Hinton used rectifiers in their 2012 ImageNet system.
Divide the illustrative scores (3,1,0) by their sum. The result is (0.75,0.25,0).
The third option gets zero chance. A negative score could give a negative chance.
Give each possible next token an exponentiated score, the value 2^{\text{score}}. One extra score point doubles this value.
The probability of each option equals its exponentiated score divided by the total. A score of -1 gives 2^{-1}=0.5, so every option receives a positive chance.
This exponential normalization is a softmax function. Adding the same value to every score multiplies every exponentiated score by the same factor.
| illustrative scores | base 2 exponentiated scores | probabilities |
|---|---|---|
| (3,1,0) | (8,2,1) | (8,2,1)/11 |
| (8,6,5) | (256,64,32)=32(8,2,1) | (8,2,1)/11 |
The common factor cancels when each exponentiated score is divided by the total, so every probability remains fixed.
Here \ln denotes a base e logarithm. For any score s, 2^s=e^{s\ln 2}. Standard softmax uses base e\approx2.718. Applying it to scores scaled by \ln 2 gives the same probabilities as the base 2 softmax.
With two base e scores, 3 and 0, softmax assigns probability 0.95 to the first option, which matches the sigmoid value from Part 3.
History: John Bridle named soft max in 1989.
In this illustrative comparison, the true chance drops from 0.51 to 0.50 and from 0.02 to 0.01. Both probability changes have magnitude 0.01. Should the loss increases be equal?
Suppose the true token receives chance 1/8. Starting from 1, halve three times: 1\to1/2\to1/4\to1/8. The halving loss for this prediction is 3.
The halving loss counts how many halvings reach the probability given to the true answer.
| true chance | halving loss |
|---|---|
| 1 | 0 |
| 1/2 | 1 |
| 1/4 | 2 |
| 1/8 | 3 |
| 0.03 | about 5 |
| 0.001 | about 10 |
Calculator check: the first loss increase is \log_2(0.51/0.50)=\log_2(1.02)\approx0.0286. The second is \log_2(0.02/0.01)=1. The ratio of the second loss increase to the first is 1/0.0286\approx35, so the second increase is about 35 times the first.
A loss computed with a natural logarithm is measured in nats. One halving equals \ln 2\approx0.69 nats.
When exactly one option is marked true, cross entropy is the negative logarithm of the probability assigned to that option.
If both classifiers assign probability 0.30 to the true token, this loss has the same value for both classifiers on that example.
History: Claude Shannon measured information with this halving count in 1948.
In an illustrative forecast for three cities, the prediction vector is (21,18,25), and the truth vector is (20,20,25).
The squared loss adds one squared error for every slot. The resulting sum is the squared distance between the vectors, as the Pythagorean theorem shows.
The second loss is 9, so it is larger by 4.
Classroom connection: the familiar arrow (3,4) has squared length 9+16=25.
With x=2, the network gives h_1=2, h_2=4, and y=10. For truth value 8, L=(10-8)^2=4, and the output rate is \frac{\partial L}{\partial y}=2(10-8)=4.
| value change | loss change | finite difference estimate |
|---|---|---|
| weight 3\to3.01 | 4\to4.0804 | 8.04, near 8 |
| x:2\to2.01 | 4\to4.2025 | 20.25, near 20 |
Loss change divided by value change is a finite difference quotient. As the change shrinks, the quotient approaches the derivative, the exact local rate. With other values fixed, this limiting rate for one value is a partial derivative. The weight partial derivatives form the gradient.
The chain rule multiplies rates along each path and adds contributions where paths meet.
Forward: With x=(1,-2), T_1x=(-1,4). The rectifier gives h=(0,4), and T_2h=4. Comparing the output 4 with the truth value 6 gives the output rate \frac{\partial L}{\partial y}=2(4-6)=-4.
1. Multiply the output rate by the columns of T_2: -4(2,1)=(-8,-4).
2. Apply the rectifier mask: (-8,-4)\odot(0,1)=(0,-4).
3. Weight from x_2 to a_2: its partial derivative is (-4)(-2)=8.
A transpose swaps rows and columns. Backward computation uses transposed tables to compute rates of change. Each activation function contributes an elementwise multiplier. For a rectifier, the multiplier is 0 or 1.
The backward sweep computes three weight derivatives. Each check changes one weight by 0.01 and recomputes the squared loss. Dividing the loss change by 0.01 gives a finite difference estimate.
| changed weight | computed derivative | finite difference estimate |
|---|---|---|
| T_1:x_2\to a_2, -1\to-0.99 | +8 | +8.04 |
| T_1:x_1\to a_1, 1\to1.01 | 0 | 0 exactly |
| T_2:h_2\to\mathrm{output}, 1\to1.01 | −16 | −15.84 |
| answer family | loss | initial loss derivative |
|---|---|---|
| number | squared loss | \frac{\partial L}{\partial \hat y}=2(\hat y-y) |
| token | base 2 softmax followed by base 2 cross entropy | \frac{\partial L}{\partial s_i}=p_i-y_i |
Here \hat y is a number prediction, s_i is one token score, and p_i and y_i are its probability and truth entry. For the illustrative token check, reuse probabilities (0.73,0.18,0.09) with truth vector (0,1,0).
| score change | finite difference estimate | p_i-y_i |
|---|---|---|
| s_1:3\to3.01 | +0.73 | 0.73-0=+0.73 |
| s_2:1\to1.01 | −0.82 | 0.18-1=-0.82 |
| s_3:0\to0.01 | +0.09 | 0.09-0=+0.09 |
For squared loss, the loss rate with respect to the prediction is twice the prediction error. For the matched softmax and cross entropy, the loss rate with respect to each score is its probability minus its truth entry. Standard base e softmax with natural logarithm cross entropy has the same score derivative.
Finite difference estimates from a 0.01 score change match the exact derivatives after rounding to two decimal places.
For an LLM batch, the forward pass computes a softmax at every training position. To form the batch loss, the training code adds or averages the halving losses. One backward sweep computes the gradient for all weights. For plain gradient descent, the negative gradient is the update direction, and the learning rate scales that direction into the weight change.
Maximum likelihood for two illustrative tokens: probabilities 1/2 and 1/4 multiply to 1/8. Their halving losses add to 1+2=3. Because logarithms convert products into sums, minimizing total halving loss maximizes the probability assigned to the training text.
Larger models and larger training corpora require more arithmetic. NVIDIA lists about one petaflop of peak dense BF16 or FP16 Tensor Core throughput for an H100 SXM. Meta reports that Llama 3 was pretrained on more than 15 trillion tokens. Because batching places many input vectors side by side, a GPU performs matrix by matrix multiplication.
The product is (5,3). The first row adds 4+1, and the second row adds 0+3.
The score gradient is (+0.8,-0.8), so its negative, (-0.8,+0.8), is the local descent direction in score space. A direct score adjustment would lower the first score and raise the second. Backward computation uses the score gradient to compute the weight gradient. For plain gradient descent, the negative weight gradient is the weight update direction, and the learning rate sets the size of the weight change.
Next: Part 6 runs training at full scale: the stages, the GPUs, and the bill. Part 9 opens attention.