Normalise within each example

A journey through normalisation layers with Botie.

Neural Networks · A journey with Botie

First visit? Learn the workshop controls

Botie's notebook

Could We Give Each Layer a More Manageable Scale?

Part 1 of 3

As earlier layers change during training, the numbers arriving at later layers change too. Their scale can make optimisation sensitive to initial weights and learning rates. We would like some control over those magnitudes inside the network, as well as at its input.

Sources for this history

Ioffe and Szegedy (2015), Batch Normalization
Ba, Kiros and Hinton (2016), Layer Normalization

← Previous lesson: Dropout Read the full lesson and derivations