Training at Scale

Every modern assistant is shaped in three stages, each running Part 5's loop on a different kind of data.

All three stages run the same training loop; the stages differ in their data and their loss.

Pretraining plays Part 1's game at planetary scale: read text, predict the next token, and update the weights after every position. Because the text itself supplies each correct next token, no person labels anything.

Each position t charges the halving loss of the true next token x_t given everything before it, and the sum runs over every position of every document. One halving is one bit; published implementations sum base e logarithms, where one halving equals 0.69 nats (Part 5's reconciliation).

One position, worked with illustrative numbers: after thecatsatonthe the model gives mat probability 0.25, so this position costs -\log_2 0.25 = 2 halvings; a sharper model that gives 0.5 pays 1 halving.

A pretrained base model continues text; asked a question, it may continue with more questions. Supervised fine tuning (SFT) teaches the assistant role from demonstrations: a prompt paired with an answer that a person wrote or approved.

The formula is Stage 1's loss restricted by a mask to the answer tokens, and the mask is Part 5's elementwise product: a zero switches a position's loss off. The model reads the prompt and is charged loss only on the answer tokens.

Documented scale: InstructGPT (Ouyang et al. 2022) tuned GPT-3 on about 13,000 demonstration prompts for 16 epochs; the paper reports validation loss overfitting after one epoch while human ratings kept improving. Tülu 3 (Ai2, 2024) mixed 939,344 demonstrations. While pretraining runs for weeks to months, this stage finishes in hours to days.

Because people compare two answers more reliably than they compose an ideal one, the training data takes a new form: one prompt, two answers, and a choice. A reward model, a copy of the LLM whose output head produces one score, learns to predict the choice.

This is the Bradley and Terry comparison model (1952), and \sigma is Part 3's sigmoid applied to a score difference. With illustrative scores: r_A=1.5 and r_B=0.5 give \sigma(1.0)\approx 0.73, so answer A should win about 73 of 100 comparisons.

The reward model trains on the loss -\log\sigma\big(r_{\text{chosen}}-r_{\text{rejected}}\big), which falls as the chosen answer's score climbs above the rejected answer's. InstructGPT trained its reward model on 33,000 comparison prompts (Ouyang et al. 2022).

The same comparison mathematics ranks chess players: Arpad Elo's rating system, adopted by FIDE in 1970, reads a rating gap as an expected score.

Proximal Policy Optimization (Schulman et al. 2017) performs the update, working with the probability ratio between the new and the previous policy, and it clips every step:

A is the advantage: how much better the answer scored than expected, where the expectation comes from a separate value network. With the paper's suggested \varepsilon=0.2, an advantage of +1 and a ratio already at 1.3 earn \min(1.3,\,1.2)=1.2, so a step past the clip range adds nothing to the objective; PPO enforces Part 4's small steps inside the loss itself.

Four LLM sized networks resident at once make PPO the most memory hungry stage; the two methods on the next slides each drop parts of this list.

Direct Preference Optimization (Rafailov et al., NeurIPS 2023) starts from the RLHF objective, solves it in closed form, and substitutes the solution back. Because the reward model and the sampling loop cancel out of the algebra, one supervised loss on preference pairs remains:

Here y_w is the chosen answer and y_l the rejected one; each log ratio measures how much the tuned model has raised that answer's probability relative to the frozen reference; the loss falls when the chosen answer's ratio climbs above the rejected answer's; and \beta sets the drift penalty exactly as it did in the RL objective.

With illustrative numbers: at the start both ratios equal 1, the bracket is 0, \sigma(0)=0.5, and the loss is one halving. When training lifts the bracket to 1.0, \sigma(1.0)\approx 0.73 and the loss falls to about 0.45 halvings.

What remains in memory: the policy and the frozen reference; no reward model, no value network, no sampling during training. Meta's Llama 3 post training ran rounds of SFT and DPO (Grattafiori et al. 2024).

DeepSeek-V3's own ledger (2024 technical report, Table 1) splits 2.788M H800 GPU hours as: pretraining 2,664K; context extension 119K; post training 5K. In that ledger, post training amounts to about 0.2% of the total (a derived share).

The loop is identical across the stages, while the compute spans three orders of magnitude.

A CPU spends its silicon on a few cores that finish one task quickly; a GPU spends it on thousands of small cores that run one instruction on thousands of numbers at once. Rasterization, filling in the pixels of each triangle, repeats the same small calculation millions of times per frame.

Matrix multiplication has the same shape of work. Multiplying two n\times n tables takes n^3 multiplications on 2n^2 input numbers. At n=1{,}000: two million numbers enter, one billion multiplications run, and every number is reused a thousand times, so the arithmetic outpaces the memory traffic and thousands of cores stay fed.

NVIDIA's Volta generation (2017) added Tensor Cores, circuits that multiply whole 4 by 4 matrices in a single clock cycle, reading FP16 inputs and accumulating in FP32. The H100 makes Part 5's scale note concrete:

Jensen Huang, Chris Malachowsky, and Curtis Priem planned NVIDIA over meetings at a Denny's in East San Jose in late 1992 and incorporated on April 5, 1993. The GeForce 256 (October 1999) shipped with the marketing line "the world's first GPU" because it moved transform and lighting, the vertex matrix work, onto the chip.

Because only graphics programs could reach the cores, researchers first phrased general arithmetic as drawing operations; Ian Buck's Brook for GPUs (SIGGRAPH 2004) systematized the approach, and Buck then led CUDA at NVIDIA, which removed the graphics phrasing altogether.

AlexNet (Krizhevsky, Sutskever, and Hinton, 2012) trained "between five and six days on two GTX 580 3GB GPUs" and won ImageNet. Because researchers across the field then moved to GPU training, NVIDIA became the first company to end a trading day valued above four trillion dollars (July 10, 2025) and the first to reach five trillion (October 29, 2025; CNBC).

Raymond James analysts estimated in 2023 that an H100 sells for $25,000 to $30,000, and shortage era retail listings reached $40,000 (CNBC reporting). At $25,000 to $30,000, the peak Llama cluster's 16,384 GPUs represent $410M to $492M of hardware before servers, networking, and buildings.

DeepSeek-V3's report states its cost figure covers "only the official training", excluding "prior research and ablation experiments on architectures, algorithms, or data". Power, derived from the model card's 700 W per GPU: the peak Llama cluster draws about 11.5 MW for the GPUs alone.

Duration derivations: 30.84M GPU hours across 16,384 GPUs give 1,882 hours, about 78 days, and longer in practice because early stages ran on 8,192 GPUs; 2.788M across 2,048 give about 57 days.

The location based figure counts what the local grid actually emitted, while the market based figure subtracts renewable energy that the company bought to match its use, which is how Meta reports 0. The 405B model alone accounts for 8,930 of the 11,390 tonnes.

An average US passenger car emits about 4.6 tonnes per year (EPA), and a transatlantic round trip seat costs about 1 tonne of CO2, roughly 2 with high altitude effects included (carbonfootprint.com calculator, 2025), so GPT-3's training equals about 120 car years, or the annual electricity of about 123 US households (EIA average, 10,500 kWh).

The 21.6 GWh line is a derivation from 30.84M GPU hours at 700 W; it covers the GPUs alone and excludes cooling and facility overhead, so the true facility figure sits higher.

Data centres of every kind, AI included, drew about 415 TWh in 2024, about 1.5% of the world's roughly 30,700 TWh of electricity; the IEA's base case reaches about 945 TWh by 2030, just under 3% (IEA, Energy and AI, 2025; Ember, 2025).

Sector comparisons need care, because the units differ: data centres above are measured as a share of electricity, while aviation contributes about 2.5% and cement about 7% of global CO2 (Our World in Data from Lee et al. 2021; IEA). The shares therefore indicate rough magnitude only.

Per use, the cost is small: about 0.3 Wh per chatbot query (Epoch AI, February 2025), which runs a 60 W bulb for 18 seconds; OpenAI's June 2025 blog states 0.34 Wh, a company figure. The older 3 Wh estimate (de Vries, 2023, Joule) predates measured efficiency gains.

One frontier training run draws about as much electricity as 2,000 US households use in a year (derived from the EIA average), while the whole data centre sector remains a small, quickly growing share of the world's grid.