0% noise
Part 10
Diffusion
Mehmet Kerem Turkcan
Associate Research Scientist
Center for Smart Streetscapes, Columbia University
New York, USA
keremturkcan.com; mkt2126@columbia.edu
Move through the deck with (i) the arrow keys, (ii) a presentation clicker, or (iii) the buttons at the bottom left. The gear at the bottom right opens the slide settings.
Several pictures from one prompt
a red kite above blue water
Question: What does each different seed change inside the sampler? It produces a different random starting tensor.
Part 9 Overview
- Three projections: each hidden row produces a query, a key, and a value through learned matrices.
- Attention weights: scaled dot products and softmax determine the weighted sum of value rows.
- Multiple heads: separate heads can gather different relationships before an output projection combines them.
- Transformer layers: residual connections and feedforward networks surround the attention calculation.
Diffusion reuses (i) image tensors from Part 8, (ii) squared error from Part 5, (iii) prompt embeddings from Part 3, and (iv) attention from Part 9.
One photograph under increasing noise
Take one photograph and add noise to it, a little, then more, until nothing of the picture survives. The noise looks like the static of an untuned television, and a computer adds any of these noise levels in microseconds.
15%
45%
75%
95%
100% noise
Question: Which direction along this strip is hard? Left to right is arithmetic anyone can run. Right to left is the hard direction, and a trained model walks it in many small steps.
Training pictures from the public web
Training needs photographs paired with sentences that describe them, and the public web already carries both, because many pictures on web pages carry written alt text. The Flickr30k collection gathered 31,783 captioned photographs from the photo site Flickr, while the builders of LAION-5B parsed 5.85 billion image and text pairs out of the alt text of crawled pages.
Because the training program adds the corruption itself, every photograph becomes endless training material: any picture, any noise level, and any random draw of noise give one exercise, the noisy picture beside the noise that produced it, whose answer the program already holds.
No person labels the noise: the captions arrive with the pictures, and the targets come from the program's own draws. One collection of captioned photographs supplies as many exercises as a training run can use.
How does independent noise combine?
Noise has a size, which statisticians call the standard deviation. Independent noise combines by squares, exactly as the sides of a right triangle do: noise of standard deviation 3 followed by noise of standard deviation 4 is distributed exactly as one draw of standard deviation 5, because 3^2+4^2=5^2.
the photograph
noise of 3 added
then noise of 4
one draw of 5
Because standard deviations combine by squares, the training program can produce any noise level with one draw of the right standard deviation, without stepping there.
One small step at a time
The network receives three inputs: the noisy picture, the step number, which records how much noise the picture carries, and the caption. It returns one output: its estimate of the noise inside the picture.
Question: Why one small step in place of one big jump? From pure noise the photograph is unguessable, because many photographs fit the noise equally well; a slightly noisy picture makes its noise a checkable target, and each later step corrects earlier errors.
A generation run
Run the loop from pure noise and a photograph condenses out of it, a little at each step.
Generation starts from a fresh random tensor, so a second run with different starting noise produces a different kite over different water, which is why one prompt with three seeds gave three different pictures.
The animation replays the recorded noise of the earlier frames in reverse order (illustrative); a trained sampler estimates the noise at every step in place of reading a record.
A video as one tensor
A video is a stack of photographs, so it is stored as one tensor with dimensions for frame, height, width, and channel. Nothing in the method changes when the tensor gains a dimension: noise is added over the whole tensor at once, standard deviations combine by the same squares, and the denoiser removes noise from the whole tensor over the same small steps.
Because the network reads every frame of the tensor together, the kite it uncovers in frame 12 matches the kite of frame 11, which is why generated motion holds together. Ho and colleagues built video diffusion this way in 2022, and OpenAI announced the video generator Sora in February 2024.
Named systems and their releases
| Date | System | What it brought |
|---|---|---|
| April 2022 | DALL-E 2 (OpenAI) | Diffusion image generation reached a broad public audience |
| July 12, 2022 | Midjourney | Opened its beta as a service reached through Discord |
| August 22, 2022 | Stable Diffusion (Stability AI) | Stability AI released the weights publicly; the model builds on the latent diffusion of Rombach and colleagues |
| February 2024 | Sora (OpenAI) | OpenAI announced diffusion video generation of clips up to one minute long |
| August 1, 2024 | FLUX.1 (Black Forest Labs) | An image generation model from Black Forest Labs, which the latent diffusion authors Rombach, Blattmann, Esser, and Lorenz founded |
The research line behind these products runs from the 2015 forward and reverse processes to the 2023 diffusion transformers.
Five ideas and their standard names
- Adding noise level by level is the forward process, a fixed schedule with no learning in it.
- The addition of squared standard deviations appears as the closed form with \bar{\alpha}_t, which the training program uses to reach any noise level in one calculation.
- The network's estimate of the noise is the prediction \hat{\epsilon}, scored by squared error against the noise that was actually added.
- The walk from pure noise back to a picture is sampling, the reverse process run from a fresh random tensor.
- The caption's influence on every step is conditioning, carried by cross attention and guidance.
Each of the five is computed with real numbers a student could recompute.
A picture as four values
| 1 | 0 |
| -1 | 0.5 |
An image tensor stores channel values in a table. This illustrative grayscale image contains four normalized values.
x_0=\begin{pmatrix}1&0\\-1&0.5\end{pmatrix}
- 1 displays as bright.
- 0 displays as middle gray.
- -1 displays as dark.
Every pixel value in this worked image is illustrative. Training tensors retain real values, even when a display clips values to its visible range.
First measured corruption
Keep 0.8 of each current value, then mix in 0.6 times a random noise value.
clean imagex_0=(1,\ 0,\ -1,\ 0.5)
noise samplen_1=(0,\ 1,\ 0,\ -1)
scaled image0.8x_0=(0.8,\ 0,\ -0.8,\ 0.4)
scaled noise0.6n_1=(0,\ 0.6,\ 0,\ -0.6)
x_1=0.8x_0+0.6n_1=(0.8,\ 0.6,\ -0.8,\ -0.2)
For the classroom: ask four students to calculate one coordinate each before revealing the result.
Why do the squared coefficients add to one?
Variance is the average of the squared distances from the mean. Start with two values whose mean is zero.
starting variance(-1,1)\quad\Longrightarrow\quad \dfrac{(-1-0)^2+(1-0)^2}{2}=1
scale by 0.8(-0.8,0.8)\quad\Longrightarrow\quad \dfrac{(-0.8)^2+0.8^2}{2}=0.64
noise variance factor0.6^2=0.36
0.64+0.36=1
When the image component and noise component each have variance one, their independent variance contributions add. Independent means that the program draws noise without using the image value. The variance statement concerns one tensor coordinate across many examples and noise draws.
Second measured corruption
At the next forward diffusion step, the program uses x_1 and draws a fresh noise sample.
current imagex_1=(0.8,\ 0.6,\ -0.8,\ -0.2)
fresh noisen_2=(-1,\ 0.5,\ 1,\ 0)
scaled image0.8x_1=(0.64,\ 0.48,\ -0.64,\ -0.16)
scaled noise0.6n_2=(-0.6,\ 0.3,\ 0.6,\ 0)
x_2=(0.04,\ 0.78,\ -0.04,\ -0.16)
The two noise arrays are illustrative draws selected for simple arithmetic.
How much clean signal remains?
Question: after two steps, what coefficient multiplies the original image x_0?
x_2=0.8(0.8x_0+0.6n_1)+0.6n_2
distributex_2=0.64x_0+0.48n_1+0.6n_2
clean coefficient0.8\times0.8=0.64
At each teaching step, multiplying the current tensor by 0.8 shrinks the original image contribution, while adding the scaled fresh noise introduces random variation.
Pause: allow four seconds after the question. Students who multiplied the two retained coefficients found the clean contribution.
Signal across repeated steps
| step | coefficient on the clean image |
|---|---|
| 0 | 1 |
| 1 | 0.8 |
| 2 | 0.64 |
| 3 | 0.512 |
| 4 | 0.4096 |
The illustrative teaching schedule uses a large variance at each step so that the shrinking clean coefficient is easy to see.
Production schedules can spread corruption across many smaller steps.
When the cumulative clean coefficient approaches zero, the final state has a distribution close to standard Gaussian noise.
Illustrative schedule. The 2020 denoising diffusion probabilistic model, or DDPM, experiments used 1,000 steps with scheduled variances from 10^{-4} to 0.02.
Symbols for one noise step
| symbol | spoken name | meaning |
|---|---|---|
| t | time index | the selected noise level |
| \beta_t | beta at time t | the scheduled variance added at this step |
| \alpha_t=1-\beta_t | alpha at time t | the retained variance factor |
| \eta_t | eta at time t | a fresh array of standard Gaussian noise values |
Uppercase T marks the final scheduled time. Standard Gaussian values cluster around zero, include positive and negative values, and have variance one in each coordinate.
One forward diffusion step
The standard symbols summarize the two calculations with four pixels.
x_t=\sqrt{\alpha_t}\,x_{t-1}+\sqrt{\beta_t}\,\eta_t
- x_{t-1} is the current image tensor.
- Multiplying x_{t-1} by \sqrt{\alpha_t} scales the retained image signal.
- The term \sqrt{\beta_t}\eta_t adds fresh Gaussian variation.
- Because the schedule fixes \alpha_t and \beta_t, forward diffusion contains no learned parameters.
In the first teaching step, \alpha_1=0.64 and \beta_1=0.36, so the two square roots are 0.8 and 0.6.
Direct access to any noise level
The cumulative product \bar{\alpha}_t multiplies the retained variance factors from step 1 through step t.
\bar{\alpha}_t=\alpha_1\alpha_2\cdots\alpha_t
The bar marks the cumulative product through time t.
\epsilon, pronounced epsilon, is one standard Gaussian array that represents the combined noise from all earlier steps.
x_t=\sqrt{\bar{\alpha}_t}\,x_0+\sqrt{1-\bar{\alpha}_t}\,\epsilon
two teaching steps\bar{\alpha}_2=0.64(0.64)=0.4096
clean amplitude\sqrt{0.4096}=0.64
Because sums of independent Gaussian values remain Gaussian, training code can draw one effective \epsilon and construct x_t directly.
One direct corruption
Consider the image with two values, x_0=(1,-1).
clean imagex_0=(1,\,-1)
selected level\bar{\alpha}_t=0.36, so \sqrt{\bar{\alpha}_t}=0.6 and \sqrt{1-\bar{\alpha}_t}=0.8
sampled noise\epsilon=(-1,\,1)
x_t=0.6(1,-1)+0.8(-1,1)=(-0.2,\,0.2)
For this example, the program retains (i) x_0, (ii) t, (iii) \epsilon, and (iv) the constructed x_t while it calculates the loss.
Illustrative image, noise level, and noise sample.
Noise prediction error
The neural network receives the noisy values and the time. Suppose its illustrative prediction is \hat{\epsilon}=(-0.5,\,0.5).
target noise\epsilon=(-1,\,1)
prediction error\hat{\epsilon}-\epsilon=(0.5,\,-0.5)
squared errors(0.25,\,0.25)
Mean squared error averages the squared errors. The symbol L names this loss.
L=\frac{0.25+0.25}{2}=0.25
Loss across a training batch
The denoiser calculates \hat{\epsilon} from (i) the noisy tensor x_t, (ii) the time t, and (iii) an optional condition c.
L=\operatorname{mean}\!\left((\hat{\epsilon}-\epsilon)^2\right)
- c names information, such as prompt features, that is paired with the clean image x_0.
- \theta, pronounced theta, names every learned parameter that influences \hat{\epsilon}.
- \operatorname{mean} averages the squared errors across tensor entries and examples in the current batch.
- For each example, the program samples a paired image and condition, (x_0,c), a time t, and a noise tensor \epsilon.
Across many batches, random sampling approximates an average over the training data, times, and Gaussian noise tensors.
Clean estimate from predicted noise
Solving the direct corruption equation for x_0 defines \hat{x}_0 from the predicted noise.
\hat{x}_0=\frac{x_t-\sqrt{1-\bar{\alpha}_t}\,\hat{\epsilon}}{\sqrt{\bar{\alpha}_t}}
plug in\hat{x}_0=\dfrac{(-0.2,0.2)-0.8(-0.5,0.5)}{0.6}
subtract\hat{x}_0=\dfrac{(0.2,-0.2)}{0.6}
estimate\hat{x}_0=(1/3,\,-1/3)
If the model predicts the exact noise (-1,1), the same calculation recovers (1,-1).
This equation calculates a clean estimate from the current state and the predicted noise.
One training example
One initialized weight
Illustrative predictor with one weight, initialized at 0.10
initial examplew=0.10,\quad x=1,\quad \epsilon=0.50
initial prediction\hat{\epsilon}=wx=0.10(1)=0.10
current lossL=(0.10-0.50)^2=(-0.40)^2=0.16
nearby loss valuew=0.11\ \Longrightarrow\ L=(0.11-0.50)^2=0.1521
Before training, software initializes the denoiser weights once from a distribution chosen by the implementation.
Two slope calculations
\Delta, pronounced delta, means change in a quantity.
two changes\Delta w=0.11-0.10=0.01,\quad \Delta L=0.1521-0.16=-0.0079
finite difference slope\dfrac{\Delta L}{\Delta w}=\dfrac{-0.0079}{0.01}=-0.79
Read \partial L/\partial w as the partial derivative of L with respect to w.
loss gradient\dfrac{\partial L}{\partial w}=2(wx-\epsilon)x=2(0.10-0.50)(1)=-0.80
The exact local slope, -0.80, is close to the finite difference slope, -0.79. Both negative values indicate that a small increase in w lowers the loss near w=0.10.
One weight update
The optimizer uses the loss gradient to update w. The illustrative learning rate r=0.05 scales the gradient and determines the update magnitude.
current valuesw=0.10,\quad x=1,\quad \epsilon=0.50,\quad \dfrac{\partial L}{\partial w}=-0.80,\quad r=0.05
w_{\mathrm{new}}=w-r\frac{\partial L}{\partial w}=0.10-0.05(-0.80)=0.14
new prediction\hat{\epsilon}_{\mathrm{new}}=0.14(1)=0.14
new lossL_{\mathrm{new}}=(0.14-0.50)^2=(-0.36)^2=0.1296
comparison0.1296<0.16, so this update lowered the illustrative loss.
During real training, the optimizer updates many weights across the denoiser's connected layers.
Training code
| PyTorch pseudocode | calculation |
|---|---|
| x0 = image_batch | read a batch of clean tensors |
| t = torch.randint(1, T + 1, (B,), device=x0.device) | choose one noise level per image |
| eps = torch.randn_like(x0) | draw the target Gaussian noise |
| a = alpha_bar[t][:, None, None, None] | look up each cumulative retained variance |
| xt = torch.sqrt(a) * x0 + torch.sqrt(1 - a) * eps | construct each noisy input directly |
| eps_hat = model(xt, t, text_features) | run the neural network forward pass |
| loss = ((eps_hat - eps) ** 2).mean() | average the squared prediction error |
| optimizer.zero_grad() | clear gradients from the preceding batch |
| loss.backward() | run the backward pass through the denoiser |
| optimizer.step() | update the learned parameters |
The code assumes that alpha_bar has an unused entry at index zero.
Four distinct operations
| operation | when it runs | what it does | learned weights |
|---|---|---|---|
| forward diffusion | training input creation | mixes a clean tensor with scheduled noise | none |
| network forward pass | training and sampling | predicts noise from the current tensor, time, and condition | reads the learned weights |
| backward pass | training | computes gradients of the loss through the network | calculates gradients for the learned weights |
| reverse diffusion | sampling | uses repeated denoiser predictions to calculate earlier states | reads the learned weights without updating them |
Teaching cue: require the full operation name whenever a student uses the word forward or backward.
Does the model replay a stored path?
Incorrect replay assumption
The system stores the noise added to one training photograph, then subtracts the same values during generation.
Question: where would that stored photograph appear when a user supplies only a prompt?
Mechanism: during each training example, the program draws a fresh time index and a fresh noise tensor. The shared denoiser learns conditional predictions across many images and noise levels.
Sampling starts from a newly drawn random tensor. Using parameters learned across many images and noise levels, the denoiser maps each current tensor, time index, and condition to a noise prediction.
Random starting noise
x_T=\begin{pmatrix}0.7&-1.1\\0.2&0.5\end{pmatrix}
Generation draws each entry of x_T from a standard Gaussian distribution.
When the forward schedule is chosen appropriately, the distribution of corrupted training images at time T is close to the standard Gaussian starting distribution.
The trained parameters stay fixed while the tensor changes through the reverse steps.
The matrix is an illustrative random draw.
Reverse mean for one value
At one illustrative time, the schedule lists \alpha_t=0.64, \beta_t=0.36, and \bar{\alpha}_t=0.36.
current valuesx_t=-0.20,\quad \hat{\epsilon}_t=-1
correction\dfrac{0.36}{\sqrt{1-0.36}}(-1)=\dfrac{0.36}{0.8}(-1)=-0.45
corrected state-0.20-(-0.45)=0.25
\mu_\theta=\frac{0.25}{\sqrt{0.64}}=\frac{0.25}{0.8}=0.3125
\mu, pronounced mu, names the predicted mean; \theta names the learned parameters. This scalar is one coordinate of that mean.
Randomness in one reverse step
At intermediate steps, an ancestral DDPM sampler adds a fresh standard Gaussian draw z.
previous product\bar{\alpha}_{t-1}=\bar{\alpha}_t/\alpha_t=0.36/0.64=0.5625
posterior variance\tilde{\beta}_t, read as beta tilde, is 0.36\,\dfrac{1-0.5625}{1-0.36}=0.2461
standard deviation\sigma_t, pronounced sigma at time t, is \sqrt{0.2461}=0.4961; the illustrative draw is z=-0.20
x_{t-1}=0.3125+0.4961(-0.20)=0.2133
At the final step, the sampler uses z=0.
Illustrative values rounded to four decimal places, using the DDPM fixed small variance \sigma_t^2=\tilde{\beta}_t.
The general reverse update
The network uses (x_t,t,c) to calculate the predicted noise \hat{\epsilon}_t. The scalar calculation below applies to every coordinate of the current tensor.
\mu_\theta(x_t,t,c)=\frac{1}{\sqrt{\alpha_t}}\left(x_t-\frac{\beta_t}{\sqrt{1-\bar{\alpha}_t}}\,\hat{\epsilon}_t\right)
x_{t-1}=\mu_\theta(x_t,t,c)+\sigma_t z
- The schedule converts that prediction into the reverse mean.
- For t>1, z is a fresh standard Gaussian tensor in ancestral sampling.
- The newly calculated x_{t-1} becomes the input to the next model call.
Sampling code
| PyTorch pseudocode | calculation |
|---|---|
| x = torch.randn(sample_shape) | draw the starting tensor |
| for t in range(T, 0, -1): | visit the noise levels in reverse order |
| eps_hat = model(x, t, text_features) | predict noise with fixed parameters |
| scale = beta[t] / torch.sqrt(1 - alpha_bar[t]) | calculate the correction scale |
| mean = (x - scale * eps_hat) / torch.sqrt(alpha[t]) | calculate the reverse mean |
| z = torch.randn_like(x) if t > 1 else 0 | draw fresh noise for intermediate steps |
| x = mean + sigma[t] * z | calculate the next tensor |
Real implementations batch model calls, manage devices and precision, and may use a sampler with a different update.
The U-Net denoiser
The 2020 DDPM denoiser used a backbone based on U-Net, with residual blocks, attention, and a sinusoidal time embedding.
Noise level conditioning
Question: should the same visible value receive the same correction at every time?
| current value | noise level | likely context |
|---|---|---|
| 0.2 | small t | much of the clean signal remains |
| 0.2 | large t | random noise contributes much of the observed value |
The model needs the noise level.
A time embedding maps t to a vector. Network blocks receive that vector so that one shared denoiser can behave differently across the schedule.
Attention from image features to text
Latent diffusion uses cross attention between image features and prompt embeddings.
prompt tokensred kite
illustrative weights(0.75,\ 0.25)
value vectorsv_{\mathrm{red}}=(1,0),\quad v_{\mathrm{kite}}=(0,2)
0.75(1,0)+0.25(0,2)=(0.75,\,0.50)
Learned projections of image features form queries, while learned projections of prompt embeddings form keys and values. The weighted sum of value vectors updates the image feature inside the denoiser.
Illustrative vectors and weights. Real prompt embeddings contain many coordinates and depend on tokenization.
Guidance from two predictions
During training for classifier-free guidance, the program sometimes replaces a prompt condition with an empty condition, so one denoiser learns conditional and unconditional predictions.
illustrative predictions\hat{\epsilon}_{\mathrm{empty}}=0.6,\quad \hat{\epsilon}_{\mathrm{prompt}}=0.2
Question: with the illustrative guidance scale s=2, what value does the sampler use?
\hat{\epsilon}_{\mathrm{guided}}=\hat{\epsilon}_{\mathrm{empty}}+s(\hat{\epsilon}_{\mathrm{prompt}}-\hat{\epsilon}_{\mathrm{empty}})
guided result0.6+2(0.2-0.6)=0.6+2(-0.4)=-0.2
At s=1, this convention returns \hat{\epsilon}_{\mathrm{prompt}}. When s is larger, the difference \hat{\epsilon}_{\mathrm{prompt}}-\hat{\epsilon}_{\mathrm{empty}} receives more weight, which can reduce output variation or create artifacts.
Latent diffusion
Rombach and colleagues, in 2022, applied the repeated diffusion calculations to an autoencoder's latent representation and used cross attention for conditioning inputs such as text.
Diffusion across video frames
In their 2022 Video Diffusion Models architecture, Ho and colleagues used spatial convolutions within each frame and temporal attention across frames.
Why do seeds change the picture?
A seed initializes a random number generator, which produces (i) the starting tensor and (ii) any later random draws.
| run settings | expected path within one implementation |
|---|---|
| same model, prompt, sampler, settings, and seed | the random number sequence repeats |
| same settings with a different seed | a different random number sequence begins |
| same seed with a changed model or sampler | the tensor sequence can differ because the model or sampler calculation changed |
Question: does the seed contain a hidden picture? The seed initializes a random number sequence. The trained denoiser maps each current tensor, time index, and prompt condition to a noise prediction.
Fewer sampling steps
Each sampling step calls the denoiser, so fewer calls usually reduce generation time.
T\longrightarrow T-1\longrightarrow\cdots\longrightarrow 1
T\longrightarrow t_3\longrightarrow t_2\longrightarrow t_1\longrightarrow 0
Song, Meng, and Ermon introduced denoising diffusion implicit models, or DDIMs, in 2021. Their update reuses a DDPM training objective and permits deterministic trajectories with selected time levels.
The resulting speed and image quality depend on (i) the trained model, (ii) the sampler, and (iii) the selected time sequence.
A deterministic DDIM trajectory still changes when its initial random tensor changes.
What does one generated image establish?
Illustrative generation record
| model | classroom denoiser A |
|---|---|
| prompt | a red kite above blue water |
| seed | 17 |
| sampler | DDIM, 30 steps |
| guidance scale | 2 |
visible claimThe pixels establish that one red kite appears above blue water.
external claimExternal records establish whether a matching real event occurred.
For the classroom: record the model version, prompt, seed, sampler, step count, guidance scale, and edits. Compare several seeds when interpreting model behavior.
The path to modern diffusion
2015Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli described a fixed forward diffusion process and a learned reverse process for generative modeling.
2020Jonathan Ho, Ajay Jain, and Pieter Abbeel presented denoising diffusion probabilistic models with a simplified noise prediction objective for image synthesis.
2021Jiaming Song, Chenlin Meng, and Stefano Ermon published DDIMs; Prafulla Dhariwal and Alex Nichol demonstrated classifier guidance for image synthesis.
2022Jonathan Ho and Tim Salimans published the full classifier-free guidance paper; Robin Rombach and colleagues published latent diffusion; Jonathan Ho and colleagues published video diffusion.
2023William Peebles and Saining Xie presented diffusion transformers, whose denoiser processes latent patches with a Transformer.
Tutor practice
1. Compute 0.8(0.5)+0.6(-0.5).
0.4-0.3=0.1.
2. With x_0=1, \bar{\alpha}_t=0.36, and \epsilon=-1, compute x_t.
x_t=0.6(1)+0.8(-1)=-0.2.
3. For \epsilon=(-1,1) and \hat{\epsilon}=(-0.5,0.5), compute mean squared error.
The two squared errors are 0.25 and 0.25, so their mean is 0.25.
4. Which training action changes the learned parameters?
The optimizer step updates the parameters using gradients calculated during the backward pass.
5. What clean image enters the sampler at time T?
Sampling begins with a fresh Gaussian tensor, so no clean image enters at time T.
Part 10 Overview
- Forward diffusion: a fixed schedule constructs noisy training inputs from clean tensors.
- Direct construction: the cumulative product \bar{\alpha}_t lets training code construct x_t directly at any selected noise level.
- Training target: the denoiser predicts sampled Gaussian noise. Mean squared error compares the predicted noise with the sampled noise so that backpropagation can calculate parameter gradients.
- Reverse diffusion: sampling begins with a fresh Gaussian tensor and repeatedly calculates earlier states with fixed model parameters.
- Denoiser inputs: a network based on U-Net receives the current tensor, the noise level, and optional conditioning information.
- Prompt control: cross attention forms context vectors from image queries and prompt keys and values, while classifier-free guidance combines empty and prompt noise predictions.
- Other tensor spaces: latent diffusion applies denoising in a compressed image representation, while video diffusion uses spatial convolutions and temporal attention across frames.
Foundational sources
- J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, S. Ganguli, 2015, “Deep Unsupervised Learning using Nonequilibrium Thermodynamics,” Proceedings of Machine Learning Research 37, pages 2256 to 2265; arXiv:1503.03585.
- J. Ho, A. Jain, P. Abbeel, 2020, “Denoising Diffusion Probabilistic Models,” Advances in Neural Information Processing Systems 33; arXiv:2006.11239. The paper defines a Gaussian forward schedule, a noise prediction objective, a denoiser based on U-Net, and an ancestral sampling procedure.
- A. Q. Nichol, P. Dhariwal, 2021, “Improved Denoising Diffusion Probabilistic Models,” Proceedings of Machine Learning Research 139, pages 8162 to 8171; arXiv:2102.09672.
- O. Ronneberger, P. Fischer, T. Brox, 2015, “U-Net: Convolutional Networks for Biomedical Image Segmentation,” Medical Image Computing and Computer-Assisted Intervention, pages 234 to 241; arXiv:1505.04597.
- J. Song, C. Meng, S. Ermon, 2021, “Denoising Diffusion Implicit Models,” International Conference on Learning Representations; arXiv:2010.02502.
- P. Dhariwal, A. Nichol, 2021, “Diffusion Models Beat GANs on Image Synthesis,” Advances in Neural Information Processing Systems 34; arXiv:2105.05233.
Conditioning and system sources
- J. Ho, T. Salimans, 2022, “Classifier-Free Diffusion Guidance,” arXiv:2207.12598. A shorter version appeared at the NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications.
- R. Rombach, A. Blattmann, D. Lorenz, P. Esser, B. Ommer, 2022, “High-Resolution Image Synthesis with Latent Diffusion Models,” CVPR, pages 10684 to 10695; arXiv:2112.10752.
- J. Ho et al., 2022, “Video Diffusion Models,” Advances in Neural Information Processing Systems 35; arXiv:2204.03458.
- W. Peebles, S. Xie, 2023, “Scalable Diffusion Models with Transformers,” ICCV, pages 4195 to 4205; arXiv:2212.09748.
- C. Schuhmann et al., 2022, “LAION-5B: An Open Large-Scale Dataset for Training Next Generation Image-Text Models,” NeurIPS Datasets and Benchmarks; 5.85 billion image and text pairs parsed from the alt text of pages crawled by Common Crawl.
- P. Young, A. Lai, M. Hodosh, J. Hockenmaier, 2014, “From Image Descriptions to Visual Denotations,” TACL 2; the Flickr30k collection of 31,783 captioned photographs gathered from Flickr.
- Product releases: OpenAI announced DALL-E 2 in April 2022 and Sora in February 2024; Midjourney opened its beta on July 12, 2022; Stability AI released the Stable Diffusion weights on August 22, 2022; Black Forest Labs, founded by R. Rombach, A. Blattmann, P. Esser, and D. Lorenz, released FLUX.1 on August 1, 2024.
- The kite and hills photographs were drawn synthetically for teaching, saved as JPEG files, and noised with the forward formula at the stated shares; the animation replays the recorded noise of the strip in reverse order. All are illustrative.
- The following materials are illustrative: (i) prompts and drawings, (ii) images with four pixels, (iii) noise arrays, (iv) teaching schedules, (v) model predictions, (vi) parameter initialization, (vii) code, (viii) attention weights, (ix) latent and U-Net tensor shapes, and (x) the generation record. Every displayed numerical result follows from the shown values, subject to the stated rounding.
- Owner: Mehmet Kerem Turkcan; Associate Research Scientist; Center for Smart Streetscapes, Columbia University; New York, USA; keremturkcan.com; mkt2126@columbia.edu.
Intro to AI, Part 10: DiffusionM. K. Turkcan, Columbia University