Autoencoders

If we ask a model to compress an input and then reconstruct it, the reconstruction tells us what the compressed representation kept. An autoencoder learns that representation by trying to reduce what is lost.

10 sections · 9 questions · 4 coding challenges

1 / 10

The arrow keys move between sections too.

What Must We Keep If We Need to Rebuild the Input?

BOTIE

Imagine describing an image with a much shorter list of numbers, then asking someone to reconstruct it from that list alone. If the description is too short to copy every pixel, it must preserve some of the structure that makes the image recognisable.

Autoencoders developed as networks trained to reproduce their inputs through an internal representation. Hinton and Salakhutdinov's 2006 work demonstrated deep networks learning low-dimensional codes that could preserve structure beyond what a linear projection captured.

The encoder produces the code, and the decoder attempts the reconstruction. A restriction, such as a narrow code or another regularising constraint, makes simple copying harder and encourages useful structure. Reconstruction supplies the training target, so separate category labels are not necessary. A code useful for reconstruction is not automatically useful for every later task, and an unrestricted network may learn little more than an identity mapping.

Sources for this history

Hinton and Salakhutdinov (2006), Reducing the Dimensionality of Data with Neural Networks

What Can We Keep When We Cannot Keep Everything?

Imagine recording two sensors that usually move together. Saving every reading from both might be wasteful if one shared value describes most of their behavior. But if the sensors disagree, a single value may lose the difference. What information should a compact representation preserve?

An autoencoder tackles this by trying to reconstruct its input from a smaller or otherwise constrained code. The reconstruction tells us whether the information kept in that code was enough for the task.

Look for structure we can reuse

When both sensor readings are similar, observations lie near a diagonal line. A position along that line can describe their shared level. The important starting point is a pattern in the data, not the assumption that all information can always be compressed.

Vertical: Sensor B

Horizontal: Sensor A

Illustrative readings that move together.

1 of 4

Train the code by asking what it can reconstruct

A learned autoencoder adjusts its encoder and decoder together to reduce reconstruction error under a constraint. The example fixes both transformations so the lost difference is easy to see.

We will then ask why reconstructing inputs does not automatically give us a reliable way to generate new ones.

Part 1. Three Components of Reconstruction

1. Give the encoder, code, and decoder separate jobs

Most models are asked to produce something different from their input, a label or a number. An autoencoder is asked to produce its own input back. That sounds pointless until the input is forced through a narrow middle on the way. What comes out the other side then shows what the middle was able to carry.

The encoder maps an observation to a representation called a latent code. The code is the information passed through the middle of the system. The decoder maps that code back into the input's original arrangement.

Take the two sensors from the opening and one observation in which they read 3 and −1. The encoder on this page keeps their average, so the code is the single number 1, and the decoder copies that number into both outputs.

Observation:     [3, −1]    two numbers
Code:            [1]        one number
Reconstruction:  [1, 1]     two numbers

Each component has one job, and none of them can do another’s. The encoder decides what to keep. The code is all that crosses the middle, so anything not in it is gone. The decoder has to rebuild every output entry from the code alone, and it never sees the observation.

For an image, the input and reconstructed output may have the same height, width, and channels. Their shapes matching does not mean their pixel values match. Reconstruction error measures the difference.

The sensor trace shows the same thing in two numbers. The reconstruction [1, 1] has the same arrangement as the observation [3, −1] and a different value in both places. The reconstruction error is the size of that difference, and Part 2 calculates it.

2. Make preserving everything difficult

An unrestricted network can learn to copy inputs without discovering a useful representation. A bottleneck limits the code width, forcing the model to choose what information it can preserve. Other autoencoders use constraints such as sparsity or corruption of the input.

Copying is easy to exhibit. Let the code be as wide as the observation, let the encoder pass both readings through unchanged, and let the decoder do the same. The observation [3, −1] becomes the code [3, −1] and comes back as [3, −1], with a reconstruction error of zero. So does every other observation. A perfect score has been reached and nothing has been learned about how the two sensors relate, because the code is the data under another name.

That is why a low error is not the goal by itself. The error is useful as a measure of what a constrained code managed to keep. Narrow the code to one number and a real question appears, which is what that one number should be.

Compression is useful when the data has structure that a compact code can capture. A smaller code does not guarantee that the preserved information will help a later classifier or search task. That must be checked separately.

For two sensors that usually agree, one shared level is such a structure. When the readings are 1 and 1, the average is 1, and copying it back loses nothing. When they are 3 and −1, the average is also 1, and the disagreement is what gets lost. The same one-number code suits the first observation and fails on the second. What a narrow code gives up is decided by the encoder, and a good encoder gives up what the data rarely does.

Part 2. Calculate a Bottleneck We Can See

3. Compress two readings into their average

Our chosen encoder averages two sensor readings. The decoder copies the average into both outputs. For an input whose readings differ, the steps are:

Input x = [3, −1]

Code z = (3 + (−1)) / 2 = 1
Reconstruction = [z, z] = [1, 1]

Squared errors = [(3 − 1)², (−1 − 1)²]
               = [4, 4]
Mean squared error = (4 + 4) / 2 = 4

The reconstructed pair retains the average but loses the difference. Moving the live slider changes that difference while holding the average constant.

offset on the live slider	input	code	reconstruction	mean squared error
0	[1, 1]	1	[1, 1]	0
0.5	[1.5, 0.5]	1	[1, 1]	0.25
1	[2, 0]	1	[1, 1]	1
2	[3, −1]	1	[1, 1]	4
−2	[−1, 3]	1	[1, 1]	4

Five settings of the live slider. The last row swaps which sensor reads high.

Every row has the same code and the same reconstruction. The decoder is handed a 1 each time and has no way to know which of these observations produced it. The error column is the only place where the five differ. It grows with the square of the offset, so a disagreement twice as large costs four times as much.

A second code entry repairs it. Let a wider encoder keep the average and also half the difference between the readings, and let the decoder add that second entry to one output and subtract it from the other. For [3, −1] the code becomes [1, 2] and the reconstruction is [3, −1], with an error of zero. The 2 in that code is exactly what the one-number bottleneck threw away.

4. Replace fixed rules with trainable transformations

Let the encoder have parameters θ and the decoder have parameters φ. Training passes an observation through both, compares the reconstruction with the original, and differentiates through the entire path.

z = encoder_θ(x)
x̂ = decoder_φ(z)

L_reconstruction = mean over examples and coordinates of (x − x̂)²

Three fixed choices scored on five readings

A loss over examples needs more than one observation, so take five. Four are the readings from the opening picture in which the sensors agree, [0, 0], [1, 1], [2, 2] and [3, 3]. The fifth is the one in which they disagree, [2, 0]. The loss averages the squared error over all ten numbers. Here it is for three one-number codes, each a fixed choice of encoder and decoder.

the encoder keeps	the decoder writes	codes for the five readings	mean squared error
the average of A and B	the code into both outputs	0, 1, 2, 3, 1	0.2
sensor A alone	the code into both outputs	0, 1, 2, 3, 2	0.4
sensor A alone	the code into A and zero into B	0, 1, 2, 3, 2	1.4

The loss ranks the three without anybody inspecting them. Averaging is the best of these because its only error is on the fifth reading, where it is off by one in each output. Keeping sensor A and copying it is off by two in one output of that reading. Writing zero for sensor B is wrong on every reading in which B is not zero.

Training does not pick from a list of three. It adjusts the numbers inside the encoder and the decoder, a little at a time, in whichever direction lowers this loss. Searching over every one-number linear code for these five readings ends at an error of about 0.1858, a little under the 0.2 that averaging scores. The best code leans slightly toward sensor A, because the one reading that disagrees has A above B. It describes the line through the origin that the five readings lie closest to, and that is the connection to principal components made at the end of this step.

Mean squared error is one possible reconstruction objective for real-valued inputs. A binary or probabilistic observation model can call for a different loss. The loss defines which reconstruction mistakes the model is encouraged to avoid.

A linear bottleneck trained with squared error is closely related to finding a low-dimensional subspace, as in PCA. Nonlinear encoders and decoders can represent more complex relationships, but they also make training and interpretation less direct.

Part 3. Reconstruction Is Not Yet a Sampling Rule

5. Reproduce the chosen transformation

The SDK class takes explicit encoder and decoder matrices. It calculates codes, reconstructed rows, and mean squared error. It deliberately does not fit them in this example.

import numpy as np
from oop_ml.numpy.modern import LinearAutoencoder

model = LinearAutoencoder(
    encoder=np.array([[0.5], [0.5]]),
    decoder=np.array([[1.0, 1.0]]),
)
result = model.respond_to(np.array([[3.0, -1.0]]))
print(result.code)
print(result.reconstructed)
print(result.mean_squared_error)

Run as written, it reports a code of 1, a reconstruction of [1, 1] and a mean squared error of 4, the figures from step 3. Given the five readings from step 4 in one call, it returns five codes and a single error averaged over all of them, 0.2. Everything the lesson has calculated so far is a reconstruction, which means it started from an observation.

6. Ask where a new code would come from

Generating means producing an example that was not in the data, and there is no observation to start from. The only way in is through the decoder, and the decoder needs a code.

Once we have a decoder, it is tempting to feed it a random code and call the output a generated example. But a plain autoencoder has not necessarily organized its code space so arbitrary random codes produce useful outputs.

A thousand codes nobody encoded

The five readings from step 4 produce codes of 0, 1, 2, 3 and 1. Those are the only codes this decoder has ever been checked on. Now pick codes by a rule that ignores the data. A common first guess is a standard normal distribution, the bell curve centered on zero with a spread of one. In one seeded run of 1,000 draws from it, 534 codes were negative.

The copying decoder accepts every one of them. A code of −1.5 comes back as the readings [−1.5, −1.5]. Nothing fails, and nothing says whether a pair of negative readings is something these sensors ever report. None of the five observations looked like that.

The reconstruction error could not have warned us. It is measured only at codes that came from real inputs, so it is the same 0.2 whatever the decoder does with codes that no input produced. A sampling rule has to say which codes to draw and how often, and reconstruction training never states one. Reusing the five observed codes would be a rule, and all it could do is replay the data.

A variational autoencoder introduces an explicit prior distribution and a training objective connecting inferred codes to that prior. That gives the next lesson a specific problem to solve: choosing codes for generation.

Put against the example above, the prior is the rule for drawing codes, stated before training. The added objective pushes the codes the encoder produces toward that rule, so the decoder is trained on the region it will later be sampled from.

Hinton and Salakhutdinov describe learning low-dimensional codes by reconstructing high-dimensional inputs. Reducing the Dimensionality of Data with Neural Networks.

Continue with Variational Autoencoders.