Calibrating an LLM judge against human verdicts

Report the true positive rate and the true negative rate of a judge (a model that scores another model's output against written rules) against human labels, not its raw agreement. Raw agreement flatters an instrument on any dataset where one verdict is much more common than the other, which is most real datasets.

The number that hides the problem

A judge is a measuring instrument. Before its numbers are used, it has to be compared against the verdicts of people who know the domain.

Galtea writes that a judge must be run against a labelled gold set (the fixed set of hand labelled examples a judge is checked against), with the gap between its verdicts and the human labels measured and the prompt iterated until the gap closes. [Galtea] That is the loop. Score the gold set, read every disagreement, change one thing, score again.

The trouble starts at the reporting step. Galtea recommends measuring a judge with accuracy, precision, recall, F1, Pearson correlation, Spearman correlation and Cohen's kappa rather than one statistic, and writes that a judge with 0.9 accuracy and 0.1 Cohen's kappa[src] is essentially guessing. [Galtea]

Keep that pairing together. The 0.9 is the number that would go on a slide. The 0.1 is the number that says the 0.9 was produced by the class balance rather than by the judge understanding anything.

Why accuracy behaves this way

Hamel Husain gives the mechanism plainly. He writes that raw agreement can be misleading when classes are imbalanced, and that human labels should be treated as ground truth with the judge's true positive rate and true negative rate reported separately. [Hamel Husain]

His worked example is a judge that always answers Pass. On a set where errors occur in 5 percent[src] of examples, that judge scores 95 percent[src] agreement while catching nothing at all.

This is not a corner case. Most production features fail on a small share of traffic. That is exactly the shape where a single agreement figure is highest and least informative, and exactly the shape a release gate is reading.

Split the number in two and the problem becomes visible:

• The true positive rate asks how many of the outputs your expert failed, the judge also failed.
• The true negative rate asks how many of the outputs your expert passed, the judge also passed.

An always-Pass judge scores perfectly on one and zero on the other. No single number does that for you.

How much labelled data before the number means anything

Agreement statistics on thin data are noise wearing a decimal point.

For validating a judge, Hamel Husain recommends about 100 examples per failure mode[src], and writes that below 60 examples the confidence intervals are often too wide to support a useful conclusion. [Hamel Husain]

That floor is a reporting rule, not just a sampling target. Where a failure mode sits under it, the honest output is the label count, not a percentage. We write the count next to every agreement figure for that reason, and the sizing of the labelled set is handled in the EV-DATA protocol alongside how many examples a dataset starts from.

Where the bar comes from

The bar a judge has to clear is a decision you make, not a fact about your feature.

Arize writes that a 75 to 90 percent[src] match between judge labels and human labels indicates strong alignment and readiness to scale, and that its pre-built evaluators achieve target precision and F-scores above 70 percent[src]. [Arize AI]

Treat that published band as a reference point to argue with, not a target to inherit. A judge gating a refund decision and a judge gating a marketing summary are held to different bars by the same company, and both bars should be written down with the date they were agreed. Whichever you pick, pair it with the two rates above rather than reporting a match rate on its own.

Calibration has a shelf life

A calibration record is a statement about a day, not a property of the setup.

Future AGI writes that a judge model can move 3 to 5 points[src] on a fixed rubric (the written rules a judge applies to decide whether an output passes) in a single month without any change on your side. [Future AGI] That range is Future AGI's published figure, not something we have measured, and the same page attributes the movement to hosted models refreshing their weights rather than to any one provider.

The practical consequence is that recalibration is scheduled rather than triggered. Nothing sends you an alert when your instrument moves. A fresh sample gets labelled by hand, the judge is rerun against it, and the agreement statistics are published with the date they were taken. That commitment sits in our plans and the procedure is the EV-JUDGE protocol.

When the judge and the labels disagree, the labels win

There is one rule that keeps the whole exercise honest. Galtea warns against updating ground truth to match model behaviour. [Galtea]

A disagreement is one of three things: a rubric item that was ambiguous, a judge prompt that needs work, or a real failure the judge missed. Each of those gets written up. None of them is fixed by editing the label to agree with the model.

Who holds the labels matters too. Hamel Husain writes that one or two key individuals usually hold the judgment that matters for an AI product, and the judge is aligned iteratively against that person's critiques. [Hamel Husain] That person works for you. We hold the process and the records.

A calibrated judge is fit to score the dataset it was calibrated on. It is not a prediction of production quality, and the limits are written down. If a judge already scores a feature you ship, talk to us about what its calibration record currently says.