Part 5

The Arithmetic of Training

Mehmet Kerem Turkcan
Associate Research Scientist
Center for Smart Streetscapes, Columbia University
New York, USA
keremturkcan.com; mkt2126@columbia.edu

Move through the deck with (i) the arrow keys, (ii) a presentation clicker, or (iii) the buttons at the bottom left. The gear at the bottom right opens the slide settings.

Parts 1 to 4 Overview

  1. The recipe: After the model computes a loss, the training algorithm computes an update direction and moves the weight vector along it. Part 4 followed one weight through one chain.
  2. The logarithm: Part 4 scored an LLM with a logarithm formula. Today we will compute that formula.
  3. The unnamed objects: Part 2 used a list of numbers of fixed length. Part 3 used weighted sums. Part 1 used a table of chances.
  4. Today: vectors and matrices name those objects. Softmax and two loss functions score predictions. The backward sweep computes one loss rate for each weight.
Rate of change check: Part 4 called this the nudge test. Change one weight by 0.01 and divide the loss change by 0.01. The quotient is a finite difference estimate of the rate of change.

Vectors: lists of numbers

Position (2,1) means two meters east and one meter up. Velocity (3,4) means that position gains three meters east and four meters up each second.

A vector is a list of numbers. A list with two slots also describes an arrow on the page.

(3,4)+(0,2)=(3,6)add matching slots
2(3,4)=(6,8)scale every slot

Part 2 called the RNN memory a list of numbers of fixed length. That memory is a vector.

eastup (3, 4) (6, 8) move east first move up second
Position and velocity in illustrative units.

History: Josiah Willard Gibbs privately printed Elements of Vector Analysis in two parts, from 1881 to 1884, for his Yale classes.

The flight table: integration tick by tick

A ball starts at (0,2) with velocity (3,4). Gravity changes the velocity by (0,-10) each second, and one tick lasts 0.2 seconds.

tickpositionvelocity
0(0, 2)(3, 4)
1(0.6, 2.8)(3, 2)
2(1.2, 3.2)(3, 0)
3(1.8, 3.2)(3, −2)
4(2.4, 2.8)(3, −4)
5(3.0, 2.0)(3, −6)
012345
Coarse, illustrative numerical integration. The parabola interpolates the six computed positions.
At tick 4: where will the ball be one tick later?

At each tick, update the position by adding velocity times the tick length. After the position update, update the velocity by adding gravity times the tick length. During training, the learning rate sets how far the weight vector moves along the update direction. The backward sweep computes the rates used to determine that direction.

For the classroom: verify tick 3 by hand. Every entry uses vector addition.

The gearbox: a table times a list

A pedal spins at 3 turns per second, and a hand crank spins at 1 turn per second. Wheel A spins at twice the pedal speed. Wheel B spins at the pedal speed plus three times the crank speed.

\begin{pmatrix}2&0\\1&3\end{pmatrix}\begin{pmatrix}3\\1\end{pmatrix}=\begin{pmatrix}6\\6\end{pmatrix}

A matrix is a table with one row per output and one column per input. For each row, multiply every entry by the matching input entry. The sum of those products is one output.

pedal: 3 crank: 1 gearbox table 2 0 1 3 wheel A: 6 wheel B: 6 2 × 3 = 6 1 × 3 + 3 × 1 = 6
Illustrative gear ratios. A zero ratio means that the corresponding input contributes zero to that wheel.

History: James Joseph Sylvester coined the mathematical word matrix in 1850.

Two gearboxes in series

Question: could one gearbox replace this pair?
\begin{pmatrix}1&2\\2&1\end{pmatrix}\begin{pmatrix}2&0\\1&3\end{pmatrix}=\begin{pmatrix}4&6\\5&3\end{pmatrix}

Feed each column of box 1, the matrix on the right, through box 2, the matrix on the left. Those two answers become the columns of the combined table.

(3, 1) box 1 box 2 (18, 18) (3, 1) combined table (18, 18) (6, 6)
Both routes use illustrative gear ratios.

When the number of outputs from box 1 equals the number of inputs to box 2, multiplying the matrices produces another matrix.

History: Arthur Cayley defined matrix multiplication in his 1858 Memoir on the Theory of Matrices so that a product represents one transformation after another.

The family of products

productshapescalculationillustrative example
row times listrow, listAdd the products of matching slots.(2,1)(3,1)=7
table times listtable, listEach row times the input list produces one output.the gearbox
elementwise productlist, listEach pair of matching slots produces one output slot.(5,2,4)\odot(1,0,1)=(5,0,4)

The elementwise product multiplies each pair of matching slots independently. Because multiplication by zero gives zero, a zero in the mask blocks that slot's contribution during the backward sweep.

Scaling check: the pedal and crank both double their speed. What happens to every output wheel?

Every output doubles, because every collected contribution doubles.

For the classroom: apply the mask (1,0,0,1) to (7,3,9,2). The surviving list is (7,0,0,2).

Part 3's neuron as a row

Part 3 computed this illustrative spam score:

2.1(1)+1.4(1)+(-3.0)(0)+(-0.5)=3.0

The weights form one row. The features form one list. The superscript \mathsf T marks the horizontal list as transposed, so the matrix reads it as a column. The bias is added after the row combines its inputs.

(2.1,1.4,-3.0)(1,1,0)^{\mathsf T}-0.5=3.0

One neuron is one row.

A layer of neurons is a matrix with a bias vector.

rowcalculationoutput
1(2.1,1.4,-3.0)(1,1,0)^{\mathsf T}-0.53.0
2(0.5,-1.0,2.0)(1,1,0)^{\mathsf T}+1.00.5

The second row and its bias are illustrative. The matrix with two rows returns (3.0,0.5).

During an embedding lookup, the model fetches one row from the embedding table. GPT-2's smallest model, published in 2019, uses 50,257 token rows with 768 numbers in each row.

Part 1's chain as a table

Call the daily table T. Columns describe today, and rows describe tomorrow. The sun column is (0.8,0.2), and the rain column is (0.4,0.6). The list (1,0) means that today is certainly sunny.

\begin{pmatrix}.8&.4\\.2&.6\end{pmatrix}\begin{pmatrix}1\\0\end{pmatrix}=\begin{pmatrix}.8\\.2\end{pmatrix}

Use tomorrow's probability vector as the next input:

\begin{pmatrix}.8&.4\\.2&.6\end{pmatrix}\begin{pmatrix}.8\\.2\end{pmatrix}=\begin{pmatrix}.72\\.28\end{pmatrix}
Question: what does the table times itself mean in weather terms?
sun rain 0.2 0.4 0.8 0.6
Illustrative daily weather chances.
T^2=\begin{pmatrix}.72&.56\\.28&.44\end{pmatrix}

T^2 gives the chances of sun and rain two days later.

Where paths merge, their contributions add: 0.8(0.8)+0.4(0.2)=0.72.

History: Andrey Markov studied chains of dependent events in 1906. Sergei Bernstein used the name Markov chain in 1926.

Reading note: some textbooks place today's state in each row. Fix the convention before comparing tables.

Why does depth need an activation function?

Linear layers only

“Deeper always means smarter.”

Suppose two layers only collect weighted sums. Could one table replace both?

Two illustrative weight matrices
M_1=\begin{pmatrix}1&1\\1&-1\end{pmatrix},\quad M_2=\begin{pmatrix}1&0\\1&1\end{pmatrix}
M_2M_1=\begin{pmatrix}1&1\\2&0\end{pmatrix}

For every input, the stack and its product matrix produce the same output. Stacked linear layers therefore reduce to one layer.

A rectifier between the tables
x=(0,1)(1,-1)\to(1,0)\to(1,1)
-x=(0,-1)(-1,1)\to(0,1)\to(0,1)

Here the stack sends x=(0,1) to (1,1) and -x to (0,1). Because any matrix that sends x to (1,1) must send -x to (-1,-1), no single matrix matches both inputs.

Once a nonlinear activation function separates the tables, the stack can represent additional functions.

History: Glorot, Bordes, and Bengio published rectifier network results in 2011. Krizhevsky, Sutskever, and Hinton used rectifiers in their 2012 ImageNet system.

From raw scores to probabilities

Division by the score total

Divide the illustrative scores (3,1,0) by their sum. The result is (0.75,0.25,0).

The third option gets zero chance. A negative score could give a negative chance.

Give each possible next token an exponentiated score, the value 2^{\text{score}}. One extra score point doubles this value.

\begin{aligned}(3,1,0)&\to(8,2,1)\\&\to(8,2,1)/11\approx(0.73,0.18,0.09)\end{aligned}

The probability of each option equals its exponentiated score divided by the total. A score of -1 gives 2^{-1}=0.5, so every option receives a positive chance.

option A 8 of 11; probability 0.73 option B 2 of 11; probability 0.18 option C 1 of 11; probability 0.09
Illustrative scores, their exponentiated values, and probabilities.

Score shifts under softmax

This exponential normalization is a softmax function. Adding the same value to every score multiplies every exponentiated score by the same factor.

illustrative scoresbase 2 exponentiated scoresprobabilities
(3,1,0)(8,2,1)(8,2,1)/11
(8,6,5)(256,64,32)=32(8,2,1)(8,2,1)/11

The common factor cancels when each exponentiated score is divided by the total, so every probability remains fixed.

Here \ln denotes a base e logarithm. For any score s, 2^s=e^{s\ln 2}. Standard softmax uses base e\approx2.718. Applying it to scores scaled by \ln 2 gives the same probabilities as the base 2 softmax.

With two base e scores, 3 and 0, softmax assigns probability 0.95 to the first option, which matches the sigmoid value from Part 3.

History: John Bridle named soft max in 1989.

Halving loss by hand

Equal drops?

In this illustrative comparison, the true chance drops from 0.51 to 0.50 and from 0.02 to 0.01. Both probability changes have magnitude 0.01. Should the loss increases be equal?

Suppose the true token receives chance 1/8. Starting from 1, halve three times: 1\to1/2\to1/4\to1/8. The halving loss for this prediction is 3.

The halving loss counts how many halvings reach the probability given to the true answer.

L=-\log_2(p_{\mathrm{true}})
true chancehalving loss
10
1/21
1/42
1/83
0.03about 5
0.001about 10

Calculator check: the first loss increase is \log_2(0.51/0.50)=\log_2(1.02)\approx0.0286. The second is \log_2(0.02/0.01)=1. The ratio of the second loss increase to the first is 1/0.0286\approx35, so the second increase is about 35 times the first.

The halving loss curve

0.021 02468 probability of the true answer
Exact halving loss for true answer probabilities from 0.02 to 1.

A loss computed with a natural logarithm is measured in nats. One halving equals \ln 2\approx0.69 nats.

When exactly one option is marked true, cross entropy is the negative logarithm of the probability assigned to that option.

If both classifiers assign probability 0.30 to the true token, this loss has the same value for both classifiers on that example.

History: Claude Shannon measured information with this halving count in 1948.

A loss for list answers

In an illustrative forecast for three cities, the prediction vector is (21,18,25), and the truth vector is (20,20,25).

error vector
(21,18,25)-(20,20,25)=(1,-2,0)
squared loss
1^2+(-2)^2+0^2=5

The squared loss adds one squared error for every slot. The resulting sum is the squared distance between the vectors, as the Pythagorean theorem shows.

Illustrative question: another forecast has error vector (0,0,3). Which forecast has the larger loss, and by how much?

The second loss is 9, so it is larger by 4.

Classroom connection: the familiar arrow (3,4) has squared length 9+16=25.

Rates of change on merging paths

x = 2 h₁ = 2 h₂ = 4 output = 10 L = 4 × 1 × 2 × 3 × 1 ∂L/∂y = 4 ∂L/∂h₁ = 4 × 3 = 12 ∂L/∂h₂ = 4 × 1 = 4 ∂L/∂x = 12 × 1 + 4 × 2 = 20
Blue shows forward values. Amber shows rates of change passed backward.

With x=2, the network gives h_1=2, h_2=4, and y=10. For truth value 8, L=(10-8)^2=4, and the output rate is \frac{\partial L}{\partial y}=2(10-8)=4.

value changeloss changefinite difference estimate
weight 3\to3.014\to4.08048.04, near 8
x:2\to2.014\to4.202520.25, near 20

Loss change divided by value change is a finite difference quotient. As the change shrinks, the quotient approaches the derivative, the exact local rate. With other values fixed, this limiting rate for one value is a partial derivative. The weight partial derivatives form the gradient.

The chain rule multiplies rates along each path and adds contributions where paths meet.

The backward sweep: tables read by columns

x₁ = 1 x₂ = −2 a₁ = −1 a₂ = 4 h₁ = 0 h₂ = 4 output = 4 1 1 2 −1 mask 0 mask 1 2 1 ∂L/∂y = −4∂L/∂h₂ = −4∂L/∂a₂ = −4
Illustrative network with one negative preactivation and one zero rectifier output.

Forward: With x=(1,-2), T_1x=(-1,4). The rectifier gives h=(0,4), and T_2h=4. Comparing the output 4 with the truth value 6 gives the output rate \frac{\partial L}{\partial y}=2(4-6)=-4.

1. Multiply the output rate by the columns of T_2: -4(2,1)=(-8,-4).

2. Apply the rectifier mask: (-8,-4)\odot(0,1)=(0,-4).

3. Weight from x_2 to a_2: its partial derivative is (-4)(-2)=8.

A transpose swaps rows and columns. Backward computation uses transposed tables to compute rates of change. Each activation function contributes an elementwise multiplier. For a rectifier, the multiplier is 0 or 1.

Local meaning: The first rectifier outputs zero for this input. A positive preactivation would produce a positive rectifier output.

Checking rates of change

The backward sweep computes three weight derivatives. Each check changes one weight by 0.01 and recomputes the squared loss. Dividing the loss change by 0.01 gives a finite difference estimate.

changed weightcomputed derivativefinite difference estimate
T_1:x_2\to a_2, -1\to-0.99+8+8.04
T_1:x_1\to a_1, 1\to1.0100 exactly
T_2:h_2\to\mathrm{output}, 1\to1.01−16−15.84
  1. Active path through the first layer: the derivative is (-4)(-2)=8; the estimate is 8.04.
  2. Inactive path: a_1 changes from -1 to -0.99, so the rectifier output and loss stay fixed.
  3. Weight in the second layer: the derivative is (-4)(4)=-16; the estimate is -15.84.
Finite difference precision: As the weight change gets smaller, each estimate approaches the derivative computed by the backward sweep.

Loss rates at the model output

answer familylossinitial loss derivative
numbersquared loss\frac{\partial L}{\partial \hat y}=2(\hat y-y)
tokenbase 2 softmax followed by base 2 cross entropy\frac{\partial L}{\partial s_i}=p_i-y_i

Here \hat y is a number prediction, s_i is one token score, and p_i and y_i are its probability and truth entry. For the illustrative token check, reuse probabilities (0.73,0.18,0.09) with truth vector (0,1,0).

score changefinite difference estimatep_i-y_i
s_1:3\to3.01+0.730.73-0=+0.73
s_2:1\to1.01−0.820.18-1=-0.82
s_3:0\to0.01+0.090.09-0=+0.09

For squared loss, the loss rate with respect to the prediction is twice the prediction error. For the matched softmax and cross entropy, the loss rate with respect to each score is its probability minus its truth entry. Standard base e softmax with natural logarithm cross entropy has the same score derivative.

Finite difference estimates from a 0.01 score change match the exact derivatives after rounding to two decimal places.

The training loop in four stations

1: forward computationtables, lists, activations 2: loss comparisonone loss number 3: backward computationtranspose, mask, sum 4: weight updatelearning rate × update direction repeat with updated weights
The update changes the weights, and the next forward computation uses the revised values.

For an LLM batch, the forward pass computes a softmax at every training position. To form the batch loss, the training code adds or averages the halving losses. One backward sweep computes the gradient for all weights. For plain gradient descent, the negative gradient is the update direction, and the learning rate scales that direction into the weight change.

Maximum likelihood for two illustrative tokens: probabilities 1/2 and 1/4 multiply to 1/8. Their halving losses add to 1+2=3. Because logarithms convert products into sums, minimizing total halving loss maximizes the probability assigned to the training text.

Larger models and larger training corpora require more arithmetic. NVIDIA lists about one petaflop of peak dense BF16 or FP16 Tensor Core throughput for an H100 SXM. Meta reports that Llama 3 was pretrained on more than 15 trillion tokens. Because batching places many input vectors side by side, a GPU performs matrix by matrix multiplication.

Check for understanding

Illustrative exercise 1. Compute \begin{pmatrix}2&1\\0&3\end{pmatrix}\begin{pmatrix}2\\1\end{pmatrix}.

The product is (5,3). The first row adds 4+1, and the second row adds 0+3.

Illustrative exercise 2. Two options score (2,0), so their base 2 softmax probabilities are (0.8,0.2). The second option is true. What is the score gradient, and which direction is locally downhill in score space?

The score gradient is (+0.8,-0.8), so its negative, (-0.8,+0.8), is the local descent direction in score space. A direct score adjustment would lower the first score and raise the second. Backward computation uses the score gradient to compute the weight gradient. For plain gradient descent, the negative weight gradient is the weight update direction, and the learning rate sets the size of the weight change.

Part 5 Overview

  1. Vectors: In the flight example, we updated position and velocity as vectors. During training, the update changes the weight vector with the same arithmetic.
  2. Matrices: Each output combines entries along one row. A composition of matrices reduces to one matrix.
  3. Depth: A nonlinear activation function can prevent that reduction, allowing a stack to represent patterns no single matrix can.
  4. Probabilities and losses: Softmax maps scores to probabilities. Classifier loss counts halvings, and regression loss measures squared distance.
  5. Backward computation: Multiplication by transposed tables computes loss rates along columns. Rectifier masks set the rate through inactive rectifiers to zero. The resulting weight derivatives form the gradient.
  6. Training: For each batch from the training text, the algorithm repeats four stages: (i) forward computation, (ii) comparison with truth, (iii) backward computation, and (iv) weight update.

Next: Part 6 runs training at full scale: the stages, the GPUs, and the bill. Part 9 opens attention.

Sources and further reading

  • J. J. Sylvester, 1850, “Additions to the Articles in the September Number of This Journal, On a New Class of Theorems, and on Pascal's Theorem,” Philosophical Magazine 37; first use of matrix for an array.
  • A. Cayley, 1858, “A Memoir on the Theory of Matrices,” Philosophical Transactions 148; matrix multiplication as composition.
  • J. W. Gibbs, 1881 to 1884, Elements of Vector Analysis; two privately printed parts for Yale classes.
  • A. A. Markov, 1906, dependent quantities; 1913, the Eugene Onegin study. S. Bernstein, 1926, “Sur les sommes de quantités dépendantes,” uses a chain of A. Markov.
  • C. E. Shannon, 1948, “A Mathematical Theory of Communication,” Bell System Technical Journal 27; base 2 information and the bit.
  • J. S. Bridle, 1989, “Training Stochastic Model Recognition Algorithms as Networks can Lead to Maximum Mutual Information Estimation of Parameters,” NeurIPS; the name soft max.
  • D. Rumelhart, G. Hinton, R. Williams, 1986, “Learning representations by back-propagating errors,” Nature 323.
  • X. Glorot, A. Bordes, Y. Bengio, 2011, “Deep Sparse Rectifier Neural Networks,” AISTATS.
  • A. Krizhevsky, I. Sutskever, G. Hinton, 2012, “ImageNet Classification with Deep Convolutional Neural Networks,” NeurIPS.
  • A. Radford et al., 2019, Language Models are Unsupervised Multitask Learners; the GPT-2 model with 117 million parameters uses 50,257 vocabulary entries and 768 embedding dimensions.
  • NVIDIA, H100 Tensor Core GPU specifications; peak dense BF16 and FP16 Tensor Core throughput for H100 SXM. Meta AI, 2024, “Introducing Meta Llama 3”; more than 15 trillion pretraining tokens.
  • Owner: Mehmet Kerem Turkcan; Associate Research Scientist; Center for Smart Streetscapes, Columbia University; New York, USA; keremturkcan.com; mkt2126@columbia.edu.
Intro to AI, Part 5: The Arithmetic of TrainingM. K. Turkcan, Columbia University