Variational Autoencoders

The encoder's distribution depends on the input. It describes plausible latent codes for that particular observation and is often called an approximate posterior. The prior is a separate distribution chosen as the source of codes before an observation is given.

In the common diagonal Gaussian setup, the encoder supplies a mean and variance for each coordinate. Diagonal means the coordinates are modeled as independent within this conditional Gaussian. Implementations often predict log variance so the variance remains positive after exponentiation.

The decoder describes how an input can be generated from a code. The reconstruction term rewards codes that help account for the observed input. The KL term measures how far the inferred code distribution is from the prior.

These objectives compete. A code that ignores the input can match the prior well but reconstruct poorly. An overly powerful decoder can also learn to ignore its latent code, a failure called posterior collapse. Merely making KL small does not establish a useful representation.

Let μ be the mean, σ the standard deviation, and ε a standard normal noise draw. Instead of asking an opaque sampler to draw directly from the encoder's Gaussian, express the sample as a transformation of independent noise.

ε ~ Normal(0, 1)
z = μ + σ ⊙ ε

Illustrative coordinate:
μ = 1,  σ = 0.5,  ε = −0.4
z = 1 + (0.5 × (−0.4)) = 0.8

The symbol ⊙ means coordinate-by-coordinate multiplication. For a fixed noise draw, the code changes smoothly with the mean and deviation, so gradients can pass from the decoder back into those parameters. This is the reparameterization trick.

Use q for the encoder's approximate posterior and p for the prior and decoder distributions. The following negative evidence lower bound is minimized. The expectation averages reconstruction likelihood over sampled codes.

L(x) = E_{z ~ q_φ(z|x)}[−log p_θ(x|z)]
       + KL(q_φ(z|x) || p(z))

For a diagonal Gaussian q and standard normal p:
KL = ½ Σⱼ [μⱼ² + σⱼ² − 1 − log(σⱼ²)]

The first term depends on the decoder's observation model. A fixed-variance Gaussian likelihood leads to a scaled squared-error reconstruction term plus constants. Other observations require different likelihood choices.

For a mean of zero and deviation of one, the approximate posterior matches the standard normal prior and this KL is zero. Changing a particular noise draw does not change KL because KL compares distributions, not sampled points.

μ² = 1,  σ² = 0.25,  log(σ²) ≈ −1.3863

KL = ½ (1 + 0.25 − 1 − (−1.3863))
   = ½ × 1.6363
   ≈ 0.8181

The SDK object exposes the reparameterized sample and exact Gaussian KL. The live example keeps the chosen draw visible so the distinction between sample and distribution is testable.

import numpy as np
from oop_ml.numpy.modern import DiagonalGaussian

posterior = DiagonalGaussian(
    mean=np.array([1.0]), deviation=np.array([0.5]),
)
print(posterior.sample(np.array([-0.4])))
print(posterior.kl_to_standard_normal)

This is the latent-distribution operation, not a trained VAE. A complete implementation must connect it to an encoder and decoder, define the reconstruction likelihood, sample during training, and propagate the resulting gradients.

After fitting the model, draw a new code from the prior and decode it. There is no input image to encode on this route. Sampling more codes produces a distribution of outputs rather than a single reconstruction.

A latent coordinate does not automatically have an interpretable meaning such as rotation or color. Those meanings depend on the data, architecture, objective, and learned representation. Inspect them instead of assigning meanings in advance.