Judge calibration and recalibration

Scope

This protocol covers the judge (a model that scores another model's output against written rules) selected in EV-SCORE to score your feature. It says how the judge is aligned to your people's verdicts before any score is published, how the alignment is reported, and how often it is checked again.

A judge is a measuring instrument. An instrument that has not been compared against human verdicts produces numbers, not evidence. We treat the judge as something that has to earn its place on every cycle, not once at setup.

Inputs

• A written rubric (the written rules a judge scores an output against), one line per pass or fail condition, in your own vocabulary.
• A gold set (the fixed set of labelled examples a judge is checked against) of your real outputs, drawn from the sampling in EV-DATA, each one labelled by hand against that rubric.
• The judge prompt, the model it calls and the settings it calls with, all recorded.
• The name of the person inside your company whose verdict settles a disagreement.
• The failure modes you already know about, so the gold set can cover each one.

Steps

1. Name the person whose judgment counts

Before anything is scored, we ask you who decides. Hamel Husain writes that in most organisations there are one or two key individuals whose judgment is crucial to the AI product, and that the judge is aligned iteratively against that person's own critiques. [Hamel Husain] That person is yours, not ours. We hold the process and the records; you hold the definition of a good answer.

2. Label a gold set large enough to support a conclusion

Labelling starts small and grows until it stops teaching you anything. Hamel Husain's heuristic is to begin with around 30 examples[src] and keep going until no new failure modes appear. [Hamel Husain] For validating the judge itself he recommends about 100 examples per failure mode[src], and writes that below 60 examples[src] the confidence intervals are often too wide to support a useful conclusion. [Hamel Husain]

We report the count next to every agreement number. If a failure mode sits under that floor, we say so and report the count instead of a conclusion. That first pass is part of what the eval baseline delivers.

3. Run the judge against the gold set and close the gap

Galtea writes that a judge must be run against a labelled gold set, with the gap between its verdicts and the human labels measured and the prompt iterated until the gap closes. [Galtea] That is the loop: score the gold set, read every disagreement, change one thing in the rubric or the judge prompt, score again. The same page recommends versioning judge prompts alongside application code, which is how we store them.

4. Report true positive and true negative rates, not raw agreement

Raw agreement flatters a judge on an imbalanced set. Hamel Husain writes that raw agreement can be misleading when classes are imbalanced, and that the human labels should be treated as ground truth with the judge's true positive rate and true negative rate reported separately. [Hamel Husain] The worked example on his page is a judge that always answers Pass and still scores 95 percent[src] agreement when errors occur in 5 percent[src] of examples.

5. Report more than one statistic

One number hides too much. Galtea recommends measuring a judge with accuracy, precision, recall, F1, Pearson correlation, Spearman correlation and Cohen's kappa rather than one statistic, and writes that a judge with 0.9 accuracy and 0.1 Cohen's kappa[src] is essentially guessing. [Galtea] Your monthly report carries the set, not the flattering member of it.

6. Agree the alignment bar before the judge scores anything

The bar is a decision, not a fact about your feature. Arize writes that a 75 to 90 percent[src] match between judge labels and human labels indicates strong alignment and readiness to scale, and that its pre-built evaluators achieve target precision and F-scores above 70 percent[src]. [Arize AI] We put that published band in front of you as a reference point, sourced on the evidence page. The bar your judge has to clear is the one you agree, written into your plan, and it is paired with the rates from step 4 rather than a match rate on its own.

A judge that clears the agreed bar is fit to score the dataset it was calibrated on. It is not a prediction about production quality, and we do not present it as one.

7. Recalibrate on a fixed cycle against fresh human verdicts

Calibration expires. Future AGI writes that a judge model can move 3 to 5 points[src] on a fixed rubric in a single month without any change on your side. [Future AGI] That range is Future AGI's published figure, not something we have measured, and the same page attributes the movement to hosted models refreshing their weights rather than to any one provider.

Because of that, recalibration is scheduled rather than triggered. We pull a fresh sample, have it labelled by hand, rerun the judge against it and publish the agreement statistics with the date they were taken. Your plan commits us to at least monthly.

8. Never move the human labels to fit the judge

When the judge and the labels disagree, the labels win. Galtea warns against updating ground truth to match model behaviour. [Galtea] A disagreement is either a rubric that was unclear, a judge prompt that needs work, or a real failure the judge missed. Each of those is written up. None of them is fixed by editing the label.

9. Record which judge version produced each score

Every score we publish carries the judge prompt version, the model identifier it ran against, the gold set version and the date of the last calibration. A score without that record is not published.

Outputs

• A versioned judge prompt, stored with its rubric and its calibration history.
• True positive and true negative rates, plus the wider statistic set, with label counts.
• The agreed alignment bar, written into your plan with the date it was agreed.
• A dated recalibration entry in each monthly drift (a change in how a feature scores when you changed nothing) report, structured under EV-REPORT.
• A written list of rubric items where your own reviewers disagree, reported rather than scored.

Limits

Version and derivation

Version 1.0, derived from G-Eval, Braintrust autoevals, FutureAGI eval drift, written 22 September 2026, by DriftCaught.