Generative Adversarial Networks

Suppose we want a model to draw handwritten digits. A classifier can say which digit an existing picture most resembles, but that does not tell a generator which pixels to produce, and a brand new picture has no labeled answer to be compared against. The generator needs feedback on the examples it creates, and a GAN learns that feedback with a second model rather than writing a rule for it by hand.

The generator transforms a random latent input into a candidate observation. A model for images could output a grid of pixel values. The random input provides variation, while the generator’s parameters determine how that variation becomes an output.

Our example removes image complexity. The input is a scalar Gaussian noise draw, and the generator adds a trainable shift. It can move the distribution left or right, but cannot change its shape or represent several distinct modes.

The live experiment makes the task that small on purpose. The real collection is 128 numbers centred near 1.5. The generator starts with a shift of −1.5, so its 128 outputs are centred near −1.5, three units to the left of where they should be. Both collections are built from the same noise draws, so they have the same spread, and the generator would reproduce the real collection exactly if its shift reached 1.5. Nothing tells it that number. It has to find it from feedback.

The discriminator takes an observation and returns a score interpreted as the probability that it came from the real collection. Its training examples mix real observations with outputs of the current generator.

The basic GAN uses source labels, real or generated, which the training procedure knows automatically. Class labels such as digit identity are optional additional information in a conditional GAN. A conditional generator and discriminator both receive the condition.

The discriminator in the live experiment is as small as the generator. It multiplies the number it is shown by one weight, adds one bias, and passes the result through a sigmoid, which turns any number into a score between zero and one. A positive weight means that larger numbers look more real to it. It starts with a weight of 0.5 and a bias of zero, which is already a rough guess in the right direction. With those values it gives the real collection a mean score of 0.667 and the generated collection a mean score of 0.329.

Now the two objectives can be stated, and they are not the same objective. The discriminator is trained to be right about sources. That means raising its score on real examples and lowering its score on generated ones. The generator is trained to be mistaken for real. That means raising the discriminator’s score on generated examples. The second row of the table is where the two collide.

This is how a second model can stand in for the rule nobody could write. The discriminator’s score is a calculation, so the generator can ask how that score would change if its output changed a little, and the answer is a direction to move in. Here the weight is positive, so a larger number earns a higher score, and the generator learns to raise its shift. That is toward the real collection, because a discriminator that is doing its job has a weight that points from the generated numbers toward the real ones.

Part 1 said what each model wants. Training needs that written as a number to make smaller, one number for each model. Both are built from the same ingredient, the negative logarithm of a probability. It is close to zero when the probability is close to one, and it grows without limit as the probability falls toward zero. So it charges a model for giving a low probability to the answer it should have given.

Let G map noise z to a generated observation and D return a real-source probability. The discriminator should give real examples high probabilities and generated examples low probabilities. This SDK uses the average of the real and generated binary cross-entropies.

L_D = −½ (E_real[log D(x)]
          + E_noise[log(1 − D(G(z)))])

L_G = −E_noise[log D(G(z))]

Read the first line one term at a time. The term log D(x) is charged on real examples, and it costs least when the real score is near one. The term log(1 − D(G(z))) is charged on generated examples, and it costs least when the generated score is near zero. The second line charges the generator on those same generated examples for the opposite outcome, which is a generated score that is not near one. E means an average, over the real examples in one case and over the noise draws in the other.

The generator uses the non-saturating objective shown in the second line. It encourages a higher real score for generated examples and provides a stronger learning signal when the discriminator initially rejects them confidently. It differs from minimizing the generator term in the original minimax expression.

d/ds log(1 − sigmoid(s)) = −sigmoid(s)       = −D
d/ds (−log sigmoid(s))   = −(1 − sigmoid(s)) = −(1 − D)

The factor of one half sets the discriminator gradient scale in this implementation. It does not combine the generator and discriminator into a single loss that both minimize.

For an illustrative zero noise draw, use the initial generator shift and discriminator parameters. Sigmoid converts the discriminator’s weighted sum into its real score.

G(z) = z + m
D(x) = sigmoid(w x + b)

z = 0,  m = −1.5,  w = 0.5,  b = 0
Generated value = 0 + (−1.5) = −1.5
Discriminator logit = (0.5 × (−1.5)) + 0 = −0.75
D(G(z)) = 1 / (1 + exp(0.75))
Generator loss for this draw = −log D(G(z))

Carrying the arithmetic through gives the two numbers the trace ends on.

D(G(z)) = 1 / (1 + exp(0.75)) ≈ 0.321
Generator loss = −log(0.321) ≈ 1.137

A score of 0.321 says the discriminator thinks this value is probably generated, and it is right. The loss of 1.137 is what the generator is charged for that.

The feedback is the slope of that loss with respect to the shift. Raising the shift by a small amount raises the generated value by the same amount. The weight of 0.5 passes half of that on to the logit, and the loss falls as the logit rises. With a learning rate of 0.1, one descent step on this single draw works out as follows.

d(loss)/d(logit) = −(1 − D(G(z))) = −(1 − 0.321) ≈ −0.679
d(logit)/d(m)    = w = 0.5
d(loss)/d(m)     ≈ (−0.679) × 0.5 ≈ −0.340

New shift = −1.5 − (0.1 × (−0.340)) ≈ −1.466

The slope is negative, so the loss falls as the shift rises, and the step moves the shift up from −1.5 to about −1.466. That is the adversarial feedback in the title of this part. No rule said the generated numbers should be larger. The discriminator’s weight said it.

The discriminator’s loss needs a real example as well as a generated one. The real collection is the noise with 1.5 added, so the real partner of this zero draw is 1.5.

Real logit = (0.5 × 1.5) + 0 = 0.75
D(x) = 1 / (1 + exp(−0.75)) ≈ 0.679

L_D = −½ (log(0.679) + log(1 − 0.321))
    ≈ −½ ((−0.387) + (−0.387))
    ≈ 0.387

The two terms are equal because the real value and the generated value sit the same distance either side of zero, so this discriminator is exactly as sure of one as of the other. Its loss is smaller than the generator’s because it is the model that is currently right.

The live experiment averages over a fixed batch of Gaussian draws, not just this zero draw. During the discriminator update, generated values are treated as fixed examples. During the generator update, the gradient passes through the fixed discriminator to the generator shift.

One full alternation at a learning rate of 0.1 then moves the discriminator’s weight from 0.5 to 0.538 and the generator’s shift from −1.5 to −1.463. The shift moved a little further than the −1.466 worked above, and the order of the alternation is the reason. The discriminator is updated first, so the generator stepped against a weight of 0.538 and not 0.5.

The training distribution is the same fixed Gaussian noise batch shifted toward the positive side. Reusing the noise keeps before-and-after comparisons from changing merely because a fresh batch was sampled.

import numpy as np
from oop_ml.numpy.modern import AdversarialGaussian

noise = np.random.default_rng(3).normal(size=128)
real = noise + 1.5
history = AdversarialGaussian(generator_mean=-1.5).train(
    real, noise, steps=30, learning_rate=0.1,
)
print(history[-1])

Each recorded state reports both losses and both mean discriminator scores after a complete alternation. A discriminator score near one half can indicate confusion, but by itself cannot prove that a rich data distribution has been matched.

Read the shift column first. It rises at every one of the thirty steps, from −1.5 to 0.081, so the generator is being pushed the right way throughout. It is also far from finished. The real collection sits at 1.5, and after thirty steps the shift is still 1.42 short of it.

Now read the two loss columns, and notice that neither tells that story. The generator loss rose during each of the first seven steps, and at step 15 it is 1.199, higher than the 1.173 it started at, although the shift had improved at every step. It rose because the discriminator was improving faster than the generator was. The discriminator’s weight went from 0.5 to 0.878 over those fifteen steps, so it had become more confident about the same generated values. From step 8 to step 30 it is the discriminator loss that rises, because the generated values are moving in among the real ones and are harder to reject. Each loss measures one model against its opponent as that opponent currently stands, and the opponent keeps changing.

The last row also carries a small warning about reading the scores. At step 30 the mean score on generated values is 0.483, close to one half, while the shift is still 1.42 away from the target.

A longer run shows something a single falling loss never would. The table below gives the state after 150 alternations at the smallest, the default and the largest learning rate the live example offers.

At 0.01 the steps are too small to get far. At 0.1 the shift has gone past the target. It reached 1.5 at step 74, kept going, and peaked at 2.012 at step 141. It can overshoot because the generator does not know where the real collection is. It follows the discriminator’s weight, and that weight was still positive when the shift passed 1.5. Once the generated values sit to the right of the real ones, larger numbers begin to look less real, the weight falls through zero, and the feedback turns around. By step 150 the weight is −0.022 and the shift has started back.

At 0.3 the same excursion happened sooner. The shift peaked at 2.007 at step 46 and had come back to 1.491 by step 150. That last row is what a settled game looks like here. Both mean scores are 0.500 and both losses are 0.693, which is not an accident.

−log(0.5) = log 2 ≈ 0.693

A discriminator that answers one half to everything is charged log 2 on every example, real or generated, and the generator is charged the same. So 0.693 is the sign that the discriminator has nothing left to go on. It is not zero, and neither loss was ever heading for zero.

A mode is a region where the data concentrates. A collection of handwritten digits has one for each digit, and a generator that draws only convincing sevens has settled on a single one of them.

Mode collapse occurs when a generator produces only a narrow subset of the data’s variety. Several attractive examples can hide that failure. A discriminator that is too weak can also give unhelpful feedback, while a rapidly improving one can make the generator’s task difficult.

A collapse is easy to miss because of how samples are usually judged, which is one at a time. Each of those sevens looks real, and nothing in a judgment of one picture counts how many kinds of picture were produced. Variety is a property of the whole generated collection, and it has to be checked on the collection.

Both warnings about the opponent appeared in step 5 in small form. The rising generator loss of the first seven steps was a discriminator improving faster than the generator. The overshoot past 1.5 was a discriminator whose advice was out of date, since its weight still pointed right after the generated values had arrived.

Our scalar family cannot demonstrate image realism or mode coverage. It does expose the two objectives and actual alternating gradients. Evaluating a full GAN requires checking the generated distribution, variety, and task-specific quality beyond its training losses.

The reason it cannot is worth spelling out. The generator adds one number to every draw, so its outputs always form a single cluster with the spread of the noise. If the real collection were two separate clusters, no shift would cover both. The discriminator is limited in a matching way. One weight and one bias can only say that larger numbers look more real, or that they look less real, so it has no way to describe real values lying on both sides of the generated ones. The last practice problem runs that case and reports what both models end up saying.