Section 7 of 12 · 15 lessons
Neural Networks
How do a collection of connected neurons learn to make a prediction? We will start with what one artificial neuron actually does: how its weights, bias, and activation function affect its output. From there, we can connect neurons and work through how examples are used to adjust their settings.
Neurons and Layers
An artificial neuron has a few basic components. Once we understand what each does, we can connect neurons and follow how a set of measurements becomes a prediction.
- 01A NeuronAn artificial neuron starts with three fundamental components: weights, a bias, and an activation function. We will work through what each contributes before putting them together to calculate an output.
- 02A Dense Layer and the Forward PassOne neuron can only do so much with an input. Connecting several neurons lets us calculate different combinations of the same measurements, then pass those outputs to another layer.
- 03The Shape GuaranteeA layer needs to know which numbers belong to which example and how they are arranged. We will follow those dimensions through a network and see what changes when an image is flattened.
Training
A network's first predictions are usually poor. We need a way to measure the mistakes, work out which settings contributed to them, and use that information to make an adjustment.
- 04Loss FunctionsBefore a network can improve, we need to tell it what counts as an error. A loss function assigns a numerical cost to a prediction, and that choice affects which mistakes the model tries hardest to reduce.
- 05BackpropagationA network can contain many weights between an input and its prediction. Backpropagation works backwards through those calculations to find how each weight contributed to a change in the loss.
- 06Training a Small NetworkWe now have a prediction, a way to measure its error, and gradients that describe possible adjustments. Training brings those steps together and repeats them over examples so the network can improve.
Regularisation and Normalisation
Improving the training score is not our only concern. We also want the network to cope with new examples and to train reliably. These methods address different parts of those problems.
- 07DropoutA network can become too dependent on particular activations during training. Randomly removing some of them makes it work with different combinations, which can help it cope better with new examples.
- 08Normalisation LayersThe values passed between layers can change in scale as a network trains. Normalisation methods control aspects of those scales, but they differ in which values they compare and what they preserve.
Other Ways to Learn and Store Patterns
A network can do more than predict a label. We will look at networks that recall damaged patterns, organise similar examples, and learn from relationships between their own inputs and outputs.
- 09Hopfield NetworksWe can often recognise a familiar pattern even when part of it is damaged. A Hopfield network stores patterns in its connections and uses repeated updates to try to recover a stable pattern from a noisy input.
- 10Restricted Boltzmann MachinesAn observation may reflect several underlying patterns we cannot see directly. A restricted Boltzmann machine uses hidden units to represent those patterns and learn a probability model of the visible data.
- 11Self-Organising MapsA large collection of measurements is difficult to picture at once. A self-organising map arranges representative examples on a small grid, trying to keep similar observations near one another.
- 12Hebbian Principal ComponentsCan a neuron learn a useful direction from examples arriving one at a time? Oja's rule adjusts its weights using the input and output, while adding a correction that prevents simple Hebbian growth from continuing unchecked.
Additional Network Layers
Different tasks need different connections. A sequence may need memory, a local pattern may need a nearby reference point, and a deep network may benefit from carrying earlier information forward.
- 13Recurrent LayersA measurement in a sequence may only make sense in light of what came before it. Recurrent layers carry a state between steps, and gates give us more control over what information is kept.
- 14Radial Basis NetworksSome patterns are useful only near particular examples. A radial basis network measures how close an input is to several reference points, then combines those local responses into a prediction.
- 15Residual ConnectionsA new layer may only need to improve a representation we already have. A residual connection carries the input forward so the layer can learn a correction rather than having to recreate the entire representation.