Section 6 of 12 · 10 lessons

Choosing and Checking a Model

A model can look good on the examples we used to choose it and still make poor predictions on new data. We need a way to compare our choices without giving them access to the answers we will use for the final test.

0 of 10 finished

Establish a Fair Comparison

A score needs a reference and an evaluation population. We will establish a simple baseline, keep related observations together, and examine what changes as more training data become available.

  1. 01What Must a Model Improve On?A complicated predictor needs a simple comparison. Before fitting one, we can measure how well a rule that always predicts a training average performs on new observations.
  2. 02Would More Examples Help?Before collecting more data or choosing a more flexible model, we can examine how training and validation errors change as the training set grows.
  3. 03Have We Really Held Out a New Example?Several rows can belong to the same greenhouse, patient, or device. If we want to predict for a new entity, a random row split can put familiar entities on both sides of the evaluation.

Choosing and Checking a Model

  1. 04PipelinesPreparing data is part of fitting a model. If a scaler learns from our test examples, the test has already influenced the result. A pipeline helps us keep those fitting steps together and inside the training procedure.
  2. 05Searching for a SettingMany model settings must be chosen before fitting. We need a fair way to compare those choices using training and validation data, while keeping the final test separate from the decision.
  3. 06Which Feature MatteredA prediction alone does not tell us which measurements the model relied on. Feature importance methods give us ways to investigate that question, although their scores need careful interpretation.

From a Score to a Decision

A model's confidence and the action we take are separate questions. We need to understand what the probabilities mean before connecting them to the consequences of a decision.

  1. 07When Should a Probability Become an Action?A probability is evidence, not a complete instruction. We need to decide how likely a sensor failure must be before an inspection is worth its cost.
  2. 08Does Eighty Percent Mean Eighty Percent?A model can rank failures well and still overstate its probabilities. We need to compare its confidence with what actually happens across similar predictions.

Know the Limits of the Result

A result depends on the observations we collected and the population we tested. We will examine sampling variation and changes in incoming data before carrying a conclusion into a new setting.

  1. 09What Changed After the Model Was Fitted?A model tested on one mix of conditions may later face another. Comparing the incoming data with the reference data helps us notice changes without pretending that every change proves an accuracy failure.
  2. 10How Much Does Our Estimate Depend on This Sample?One average hides how it might change if different observations had been collected. Resampling the observed data gives us a way to study that dependence under explicit assumptions.