Diffusion Models

Adding noise to an example is easy to control. Learning to remove it gives us a useful training problem, and a trained reverse process can then turn a noisy starting point into a generated example.

10 sections · 9 questions · 4 coding challenges

1 / 10

The arrow keys move between sections too.

Could Learning to Remove Noise Teach a Model to Generate?

BOTIE

Creating a detailed image in one operation is a difficult task. Adding a little noise to an existing image is easy, and learning to account for a small corruption gives us a more manageable question. What if we could organise generation as many such steps?

Sohl-Dickstein and colleagues' 2015 diffusion work modelled data through a gradual noising process and a learned reverse process. Ho, Jain and Abbeel's 2020 work developed a particularly effective denoising formulation for image generation.

During training, we corrupt examples by known amounts and teach a model to estimate the noise or an equivalent denoising quantity. During generation, we start with noise and repeatedly apply learned reverse steps. Each step is conditioned on where we are in the process. This does not recover a uniquely hidden original image from arbitrary noise; it samples a new example using regularities learned from the training data.

Sources for this history

Can Learning to Remove Noise Teach a Model to Create?

Imagine adding a little random interference to a simple picture, then adding more until its structure is difficult to see. Creating corrupted examples is easy because we control the noise. Could those examples teach a model how to move in the other direction?

Diffusion models use this idea to construct a learning problem. We start with real examples, create noisy versions, and train a model to estimate information needed to remove noise. Generation then begins with random noise and repeatedly uses a learned reverse process.

Begin with a recognizable signal

Take a small picture with a bright region against a dark background. Its pixel values are the clean example we want the training process to preserve information about. We know those values when preparing training data.

Clean signal

Illustrative pixel grids, not outputs of a trained diffusion model.

1 of 4

Separate known training noise from a model's estimate

The example supplies a controllable fraction of the exact noise as an oracle. It shows why an accurate estimate matters and why heavy corruption can amplify estimation errors.

It does not train a denoiser or run a reverse sampler. The equations below identify where those components belong.

Part 1. Construct the Supervision Ourselves

1. Define a forward corruption schedule

Supervised learning needs inputs paired with correct answers. For generating pictures nobody can supply the answer, because there is no single correct picture to produce. Diffusion gets its answers another way. It damages clean examples itself, in a way it can describe exactly, and keeps a record of the damage. The record is the answer.

A forward step reduces the previous signal slightly and adds Gaussian noise. The schedule sets how much noise enters at each step. After enough corruption, the resulting distribution can become close to a simple Gaussian distribution.

The example on this page is a strip of six pixels, two dark, two bright and two dark, written [0, 0, 1, 1, 0, 0]. Its schedule has twenty steps of equal strength. The first step multiplies the strip by about 0.9487 and adds a Gaussian draw scaled by about 0.3162, which barely changes it. The damage comes from repetition.

forward step	multiplier on the clean strip	multiplier on the noise
0	1	0
1	0.9487	0.3162
5	0.7684	0.6399
10	0.5905	0.8070
20	0.3487	0.9372

What the strip is made of after a given number of steps. Part 2 derives both columns from the schedule.

The model does not need to learn this forward process because we specify it. Its purpose is to supply noisy inputs with known relationships to clean examples. The small schedule on this page demonstrates the relationship but is not long enough to imply that all signal is gone.

The last row of the table is that caveat in numbers. After all twenty steps the clean strip still enters with a multiplier of 0.3487, so the bright pixels are faint and not gone. The same per-step noise would have to run for 44 steps before the retained fraction of the signal, which is the square of that multiplier, fell below one percent.

2. Train a predictor that knows the noise level

A common denoising network reads the noisy sample and the timestep, then predicts the noise used in its construction. A timestep embedding gives the network information about how much corruption to expect.

One training example on this page looks like this. The clean strip and one noise draw are combined at step 5. The network is shown the noisy strip and the number 5, and the answer it is graded against is the draw.

pixel	clean value	noise draw, the target	noisy value at step 5, the input
1	0	−0.6518	−0.4171
2	0	−0.1747	−0.1118
3	1	1.6637	1.8331
4	1	0.6591	1.1902
5	0	−1.6414	−1.0504
6	0	−0.0052	−0.0033

The live example at its default step. The clean column is used to build the input and is then put away.

Nobody labeled anything. The target column came from a random number generator a moment before the input was built from it. A new draw or a new step makes a new training example from the same clean strip, so one clean example supplies as many pairs as we care to construct.

The step has to be an input because the noisy values do not reveal it. On this draw the third pixel reads 1.8331 at step 5 and 1.9080 at step 20, nearly the same number. At step 5 the clean signal entered with a multiplier of 0.7684 and at step 20 with 0.3487, so those two similar readings contain very different amounts of noise. A predictor that was not told the step could not know which case it was looking at.

A U-Net can serve as this predictor because it combines broad image context with fine spatial detail. Its diffusion target is the known added noise. This differs from the semantic segmentation task, where U-Net is trained with labeled images and corresponding target masks. Conditional diffusion can also use text or class labels.

Part 2. Calculate a Noisy Training Pair

3. Jump directly to a chosen forward step

Let β be the per-step noise variance, α the retained fraction for that step, and ᾱ the cumulative retained fraction. Independent Gaussian additions can be combined into one draw, so we can sample a chosen training step without simulating every preceding step.

αₜ = 1 − βₜ
ᾱₜ = ∏ₛ₌₁ᵗ αₛ

ε ~ Normal(0, I)
xₜ = √ᾱₜ x₀ + √(1 − ᾱₜ) ε

For this example's constant schedule at step 5:
ᾱ₅ = (1 − 0.1)⁵ = 0.59049

The square-root coefficients control the signal and noise scales. The clean sample is x₀, the noisy sample is xₜ, and ε is the known noise draw. At step zero, cumulative retention is one and the signal is unchanged.

Step 5 for two of the six pixels

The cumulative retention at step 5 gives both coefficients. The third pixel is bright and its draw was 1.6637. The first pixel is dark and its draw was −0.6518.

√ᾱ₅ = √0.59049 ≈ 0.7684
√(1 − ᾱ₅) = √0.40951 ≈ 0.6399

Third pixel:  x₅ = (0.7684 × 1) + (0.6399 × 1.6637) ≈ 1.8331
First pixel:  x₅ = (0.7684 × 0) + (0.6399 × (−0.6518)) ≈ −0.4171

Those are the values in the table in step 2, and the other four pixels follow the same line of arithmetic. The two coefficients are tied together. Their squares are ᾱ and 1 − ᾱ, which always total one, so whatever scale the signal gives up the noise takes.

step	cumulative retention ᾱ	signal coefficient √ᾱ	noise coefficient √(1 − ᾱ)
0	1	1	0
1	0.9	0.9487	0.3162
5	0.5905	0.7684	0.6399
10	0.3487	0.5905	0.8070
20	0.1216	0.3487	0.9372

The constant schedule of this page, where every step has a noise variance of 0.1.

The jump is what makes training practical. Building a training example takes one clean sample, one step number and one draw, and the noisy input comes out of a single line of arithmetic, however late the step is.

4. Predict noise and estimate the clean sample

Write the network's noise estimate as ε̂. A common simplified training loss compares the predicted noise with the actual draw. Once an estimate is available, the forward expression can be rearranged to estimate the clean sample.

L_simple = E_{x₀,t,ε}[‖ε − ε̂_θ(xₜ, t)‖²]

x̂₀ = (xₜ − √(1 − ᾱₜ) ε̂) / √ᾱₜ

If the estimate equals the exact noise, this recovers the clean sample. During generation that exact noise is unavailable. A predictor must infer useful information from the noisy input and what it learned from training data.

As signal retention becomes small, division by its square root amplifies noise-estimation errors in this clean-sample estimate. The live oracle control makes that dependence visible.

The estimate with all, half and none of the noise

The live control supplies a chosen fraction of the true draw as the noise estimate. Follow the third pixel at step 5, where the noisy value is 1.8331, the true draw is 1.6637 and the clean value is 1.

All of the draw:   x̂₀ = (1.8331 − (0.6399 × 1.6637)) / 0.7684 ≈ 1.0000
Half of the draw:  x̂₀ = (1.8331 − (0.6399 × 0.8319)) / 0.7684 ≈ 1.6928
None of it:        x̂₀ = (1.8331 − 0) / 0.7684 ≈ 2.3855

The inputs shown are rounded and the results were calculated from the unrounded values. With the whole draw the clean value comes back. With half of it the estimate is too high by 0.6928, and with no estimate at all it is too high by 1.3855. The size of the miss depends on the step as well as on the estimate, which the next lines make exact.

x̂₀ − x₀ = √((1 − ᾱₜ) / ᾱₜ) × (ε − ε̂)

An error in the noise estimate is multiplied by that square root on its way into the clean estimate. Hold the estimate at half of the true draw and move the step.

step	cumulative retention ᾱ	error multiplier	third pixel’s clean estimate
1	0.9	0.3333	1.2773
5	0.5905	0.8328	1.6928
10	0.3487	1.3667	2.1369
20	0.1216	2.6880	3.2360

The clean value is 1 in every row, and the noise estimate is wrong by the same amount in every row, half of the draw.

The same mistake about the noise costs 0.2773 at step 1 and 2.2360 at step 20. Early in the schedule the multiplier is below one and an imperfect predictor still gives a fair picture of the clean sample. Late in the schedule the multiplier is well above one, and a single leap from a very noisy sample to a clean one magnifies whatever the predictor got wrong.

Part 3. What the Reverse Process Adds

5. Distinguish clean estimation from a reverse sampling step

A full diffusion sampler starts from noise and repeatedly predicts a less corrupted state. In the DDPM formulation, one reverse update has the following form. The reverse variance σ² depends on the chosen schedule or model; ξ is fresh standard Gaussian noise and is set to zero on the last step.

μ_θ(xₜ,t) = (1 / √αₜ)
             [xₜ − (βₜ / √(1 − ᾱₜ)) ε̂_θ(xₜ,t)]

xₜ₋₁ = μ_θ(xₜ,t) + σₜ ξ

This is a sequence of learned reverse transitions, not repeated use of the exact forward noise. Different samplers change the update rule and number of steps. The page's numerical object implements forward noising and clean estimation only.

One reverse update beside one clean estimate

Both formulas take a noise estimate, and they do different things with it. Take the third pixel at step 5 again, where the noisy value is 1.8331, and give both formulas the exact draw of 1.6637, so that neither is held back by a poor prediction. On this schedule β₅ is 0.1 and α₅ is 0.9.

Clean estimate:  x̂₀ = (1.8331 − (0.6399 × 1.6637)) / 0.7684 ≈ 1.0000

Reverse mean:    μ = (1 / √0.9) × [1.8331 − ((0.1 / 0.6399) × 1.6637)]
                   ≈ 1.0541 × [1.8331 − 0.2600]
                   ≈ 1.6582

The clean estimate goes all the way to 1 in one move. The reverse update takes the pixel from 1.8331 to about 1.6582, a short distance in the right direction, and then adds fresh noise scaled by σ unless it is the last step. In the formulation above, a sampler repeats that update once for each step of the schedule, which would be twenty times here. It asks the predictor for a new estimate every time, because after each move the sample is a different noisy input at a different step.

Step 4 showed the cost of the alternative. A single leap from step 20 to a clean estimate multiplies the predictor’s error by more than two and a half. The reverse mean above was worked by hand from the formula, with the exact draw standing in for a prediction. Nothing on this page runs a reverse sampler.

6. Reproduce the oracle check

Passing the actual added noise into the recovery method is a useful correctness check. It is not a generative shortcut, because it relies on information that we know only while constructing the training example.

import numpy as np
from oop_ml.numpy.modern import GaussianDiffusion

process = GaussianDiffusion(np.full(20, 0.1))
clean = np.array([0.0, 0.0, 1.0, 1.0, 0.0, 0.0])
noise = np.random.default_rng(4).normal(size=6)
noisy = process.add_noise(clean, noise, step=5)
recovered = process.recover_clean(noisy, noise, step=5)
print(recovered)

Run as written, the code returns the clean strip, [0, 0, 1, 1, 0, 0], to within rounding in the sixteenth decimal place. Replace the true draw with zeros, which is what a predictor that had learned nothing might offer, and the same call returns [−0.5428, −0.1455, 2.3855, 1.5489, −1.3669, −0.0043]. Every pixel is wrong, and the pixels whose draws were largest are wrong by the most.

The check passes only because the script still holds the draw it added a line earlier. When generating there is no clean strip, so no draw was ever added to one, and there is nothing to pass in. The only possible source of a noise estimate is a trained predictor.

To build a working generator around these operations, we would need a trained timestep-conditioned predictor and a reverse sampler. Image quality then depends on that learned model, the training distribution, conditioning, and the sampling procedure.

The paper develops the noise-prediction training formulation and reverse sampling process used in this lesson. Denoising Diffusion Probabilistic Models.

Continue with Evaluating Generative Models.