Can a good average hide repetition?

A journey through evaluating generative models with Botie.

Adapting and Evaluating Modern Models · A journey with Botie

First visit? Learn the workshop controls

Botie's notebook

How Do We Judge an Answer When Several Answers Could Be Good?

Part 1 of 3

A classifier often has a specified label to compare with its prediction. A generated sentence or image may have many acceptable forms. Exact agreement with one reference can reject a good alternative, while a few attractive examples can hide repetition or failures elsewhere.

Sources for this history

Papineni and colleagues (2002), BLEU: A Method for Automatic Evaluation of Machine Translation
Heusel and colleagues (2017), GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

← Previous lesson: Retrieval-Augmented Generation Read the full lesson and derivations