Section 4 of 12 · 11 lessons
Data Preparation
A model only sees the records we give it. Before fitting, we need to know what a row represents, which measurements are missing, and how categories, units, and time should be represented. We also need to keep information from the future out of a prediction about the present.
Can We Trust These Records?
A blank reading, a repeated import, and an unusual observation are different problems. We need to establish what each record represents before deciding how to prepare it.
- 01What Does One Row Represent?Two identical-looking rows might be a duplicated import or two genuine observations. We need to establish a record's identity before cleaning a dataset or trusting its counts.
- 02When a Measurement Is MissingAn empty thermometer reading does not mean the greenhouse was at zero degrees. We need to distinguish what was measured from what we supplied to keep a calculation working.
Centres and Scales
A difference of ten means something very different when we are measuring age rather than income. We will look at how changing a measurement's reference point and units changes the comparisons a model makes.
- 03Centring on the MeanSometimes it is more useful to know how far a measurement is from the average than how far it is from zero. Subtracting the mean gives us that new reference point.
- 04The Standard ScoreHow unusual is a measurement that is ten units above average? That depends on how much the measurements usually vary. A standard score expresses the difference in units of standard deviation.
- 05Feature ScalingA distance calculation can be dominated by a feature simply because its numbers are larger. We will look at how scaling changes that calculation and why some models are affected more than others.
Constructing and Reducing Features
Sometimes the useful relationship is easier to describe with a new measurement, such as area rather than length. Other times we have several measurements describing much the same thing. Both affect what we give the model.
- 06Polynomial FeaturesA straight line cannot describe every relationship between our measurements and the answer. Adding features such as an input squared gives the model more to work with while keeping its coefficients straightforward to fit.
- 07Principal Component AnalysisSeveral measurements may vary together, so storing all of them can repeat much of the same information. Principal component analysis looks for directions that preserve the most variation with fewer coordinates.
Representing What We Measured
A material name is not a quantity, and an hour of the day repeats. Our representation should preserve those meanings instead of inventing relationships between arbitrary numbers.
- 08Categories Without an Invented RankingA roof can be glass, plastic, or mesh. Assigning those names arbitrary numbers can accidentally suggest a ranking. We need a representation that preserves category membership without inventing a numerical relationship.
- 09When the End Is Beside the BeginningThe hour after 23 is zero, yet ordinary subtraction puts those hour numbers far apart. A repeating feature needs a representation that keeps its beginning beside its end.
Keep Future Answers Out of Training
Our fitting procedure must respect what was available at prediction time. We also need to keep rare outcomes visible so an apparently successful score does not hide the mistakes that matter.
- 10What Could We Know at Prediction Time?A model can appear remarkably accurate when a feature contains information that arrives after the decision. We need to reconstruct what was actually available when each prediction would have been made.
- 11When the Important Class Is RareIf almost every sensor is healthy, predicting healthy every time can look accurate while missing every failure. We need to inspect what the score says about each class.