How evaluation work is actually done: building a golden dataset, choosing scorers, calibrating a judge and setting thresholds that gate a release.
How the work is actually done: datasets, scorers, judges, thresholds.
6 posts
Reference based scoring needs an expected result stored next to the input. When none exists, what replaces it is a written rubric, reference free scorers and a labelled sample your own expert graded.
The thing that ages an eval dataset is usually not a model release. It is your own users, asking things the set was never built from, while every run keeps passing.
A gate that blocks every deploy gets switched off within a fortnight. Two tiers, a delta rule and a written baseline are what keep it useful and trusted.
Take the cheapest scorer that can actually catch the failure mode. Here is what each of the three main kinds sees, what each one misses, and how to decide per failure mode.
A judge that agrees with your labellers nine times out of ten can still be guessing. Here is how published practice suggests you find out, and what one number cannot tell you.
Published guidance gives three different starting sizes for a first eval dataset. Here is what each one says, why they differ, and what actually decides your number.