Section 12 of 12 · 5 lessons

Adapting and Evaluating Modern Models

A trained model may be capable of many things without doing the particular job we need. We can teach it from examples, give it feedback, or provide information it did not have during training. We also need to check whether those changes actually make it more useful.

0 of 5 finished

Adapting a Trained Model

We may already have a useful model but need it to respond differently for a particular task. Examples, smaller parameter updates, and rewards offer different ways to teach that behaviour.

  1. 01Supervised Fine-TuningA model that can continue text may not respond in the way a task requires. Fine-tuning on examples of prompts and desired responses gives it demonstrations of the behaviour we want.
  2. 02Low-Rank AdaptationAdapting every weight in a large model can be expensive. LoRA keeps the original weights fixed and learns a smaller correction through two narrow matrices, reducing the number of parameters that need updating.
  3. 03Reinforcement LearningWe cannot always provide the correct answer before an action is taken. Reinforcement learning uses the rewards that follow actions to improve a policy, including decisions whose consequences appear several steps later.

Grounding and Evaluation

An answer can sound convincing and still be wrong. We need to provide relevant information when the task requires it and evaluate the answer against what the user actually needed.

  1. 04Retrieval-Augmented GenerationA model may need information that was absent or out of date when it was trained. Retrieval supplies relevant source material as context, giving the model evidence it can use when answering a question.
  2. 05Evaluating Generative ModelsA fluent answer or convincing picture is not enough to show that a model does its job well. We need evaluation examples and criteria that reflect the task, including failures an overall average might conceal.