Section 5 of 12 · 21 lessons
Classical Machine Learning
Different problems give us different information to work with. We might have examples with known answers, observations we want to group, or a pattern we need to find in a picture. We will work through the methods developed for these problems and why their different approaches can be useful.
Distance
Before we ask a model to find similar examples, we need to decide what similar means. Different ways of measuring distance can give us different answers.
Regression
A single straight line will not explain every relationship. We can include more measurements or allow a more flexible rule, but we still need to work out how to fit it and when that flexibility stops helping.
- 02Multiple & Polynomial RegressionOne measurement may not be enough to explain the outcome. We can give a regression model several inputs, including transformed inputs, and work out how much each contributes to the prediction.
- 03Fitting by WalkingRather than solving for a fitted line all at once, we can improve an initial guess through small adjustments. The gradient tells us which direction increases the error, so we can try moving the other way.
- 04Ridge & LassoA flexible model can fit accidental details in the training data. Penalising large coefficients gives us a way to discourage that behaviour, with ridge and lasso making different kinds of adjustment.
Classification
Sometimes we want a category rather than a number. We will look at how nearby examples, a sequence of questions, and scores for different classes can each give us a way to decide.
- 05k-Nearest NeighboursIf similar examples tend to have similar answers, nearby labelled examples may help us predict a new one. We still need to decide how to measure distance and how many neighbours to ask.
- 06Decision TreesA series of simple questions can narrow down a decision. A decision tree learns which questions to ask from the examples and places a prediction at the end of each route.
- 07More Than Two ClassesSome problems have more than two possible answers. We need a way to compare the evidence for each category and decide how the resulting scores become a prediction.
Ensembles
One model may depend too heavily on a few examples or miss a pattern another model captures. Combining models can help, but the way we train and combine them matters.
- 08BaggingSmall changes in the training data can produce very different fitted models. Fitting several models to resampled data and combining their answers can make the result less dependent on one particular fit.
- 09Random ForestsA collection of trees is less useful if they all make the same mistakes. Giving them different samples and different feature choices encourages the variety that can make their combined prediction more reliable.
- 10Gradient BoostingA model's mistakes tell us what it has not yet explained. Boosting adds models in stages, with each new model learning a correction to the predictions already made.
Clustering
We do not always have labels telling us which examples belong together. Clustering tries to find groups from the measurements themselves, using a chosen definition of similarity.
Kernels
A pattern that is difficult to separate in its original coordinates may become simple after a transformation. Kernels let us use comparisons in that transformed space without explicitly calculating every new coordinate.
- 12The Kernel TrickA pattern may be difficult to separate with a straight boundary until we change how it is represented. A kernel lets us compare examples in a transformed space without explicitly building every new feature.
- 13Kernel Ridge RegressionRidge regression gives us a way to control a fit, but its original features may not describe the relationship well. Kernel comparisons let us apply the same idea in a transformed feature space.
- 14Kernel Principal ComponentsThe useful structure in a dataset may not lie along a straight direction in its original coordinates. Kernel PCA looks for variation after a transformation, using comparisons between examples to do the work.
Sequences
The order of observations can matter. We will start with a model that uses the current state to predict the next one, so we can see what this limited form of memory allows.
Classical Computer Vision
A computer receives a picture as an array of pixel values. We need ways to turn those values into useful observations, such as where an edge lies or whether a familiar shape is present.
- 16Filters and EdgesThe boundary of an object often appears as a change in brightness. Small image filters compare nearby pixels so we can measure those changes instead of treating each pixel in isolation.
- 17Template MatchingIf we already know what a pattern looks like, we can search for it in a larger picture. Template matching compares that known pattern with each possible location in the image.
- 18Histogram of Oriented GradientsAn object's outline can remain recognisable when the lighting changes. Histograms of oriented gradients describe local edge directions, giving a detector information about shape without relying on exact pixel values.
- 19Keypoints and DescriptorsTwo photographs may show the same scene from different positions. We need distinctive points we can recognise in both pictures and a numerical description that lets us compare their surroundings.
- 20Image AlignmentA camera can move or turn between photographs. Matching points gives us evidence of that movement, which we can use to work out how to bring the images into alignment.
- 21Haar CascadesSearching every part of a picture can be expensive, especially when most regions contain nothing relevant. A cascade uses inexpensive checks to reject unlikely regions before applying more demanding tests.