Our judge says pass and our users say no

What is actually happening

A judge (a model prompted to score outputs against written rules, calibrated against human labels) does not have an opinion about your users. It applies the rules it was given to the cases it was shown, and a disagreement means one of those two things is wrong. The pass rate you are looking at cannot tell you which, and it may be hiding the problem rather than reporting it.

Start by distrusting accuracy on its own. Hamel Husain writes that raw agreement can be misleading when classes are imbalanced, and that the human labels should be treated as ground truth with the judge's true positive rate and true negative rate reported separately; his worked example is a judge that always predicts Pass and still scores 95 percent agreement when errors occur in 5 percent of examples.[Hamel Husain] Galtea makes the same point with a different statistic, recommending accuracy, precision, recall, F1, Pearson correlation, Spearman correlation and Cohen's kappa rather than one number, and writing that a judge with 0.9 accuracy and 0.1 Cohen's kappa[src] is essentially guessing.[Galtea] A judge in that state passes almost everything and reports a healthy number while doing it.

There are three candidate causes and they need different fixes. The rubric (the written rules the judge applies, in your own words, agreed with your expert) may not say what your experts actually mean, which is the most common one and the most fixable. The judge may have moved: Future AGI writes that a judge model can move 3 to 5 points[src] on a fixed rubric in a single month with no change on your side, a range Future AGI attributes to hosted frontier models refreshing weights.[Future AGI] That is Future AGI's published figure, not ours, and we have reproduced nothing of the kind. Or your golden dataset (a fixed set of labelled inputs and expected outputs, versioned, that every run scores against) may simply not contain the cases users are complaining about, in which case the judge was never asked the question.

The fourth possibility is the uncomfortable one, and it is worth saying out loud. Your users may be right about something the rubric deliberately excludes, because someone decided it was out of scope months ago. That is a product decision surfacing as a measurement complaint, and no amount of recalibration resolves it.

What to do this week

The order exists to stop you tuning the judge before you know what the right answer is. Days one and two produce human ground truth on the exact cases in dispute, with a written reason on every verdict, because the reasons are the material for the rubric rewrite on day five. Hamel Husain writes that in most organisations one or two key individuals hold the judgment that matters to the product, and that a judge is aligned iteratively against that person's critiques.[Hamel Husain] That person's labels are the ground truth here, and ours are not.

Day three compares verdict by verdict rather than in aggregate, because the aggregate is what misled you in the first place. Day four is the reporting discipline: true positive rate and true negative rate separately, and at least one statistic beyond accuracy, so the shape of the disagreement is visible. Arize's published bar for alignment is a 75 to 90 percent[src] match between judge labels and human labels.[Arize AI] Size matters before any of those numbers mean much: Hamel Husain recommends about 100 examples per failure mode[src] and writes that below 60 the confidence intervals are often too wide to support a useful conclusion.[Hamel Husain]

Day five changes the rubric first and the judge prompt second, in that order, and versions both. Galtea writes that a judge must be run against a labelled gold set, with the gap between its verdicts and the human labels measured and the prompt iterated until the gap closes.[Galtea] The judge protocol is this week written out in full, and the dataset protocol covers the case where the answer is coverage rather than calibration.

What happens without evals

The published case is a provider's own, and it is the clearest sourced example of the exact gap you are describing. In a postmortem published on 17 September 2025, Anthropic wrote that its evaluations simply didn't capture the degradation users were reporting, and that it had relied too heavily on noisy evaluations.[Anthropic] Anthropic's stated causes were three overlapping infrastructure bugs, a context window routing error, output corruption and a TPU miscompilation, not a change in model weights, and we make no claim that we would have caught it.

The lesson is about the measurement rather than the incident. A team with a real eval suite, real engineers and every incentive to find the problem had a measurement that said one thing while users said another, and the measurement was the thing that was wrong. A pass rate is not evidence of quality. It is evidence about a rubric, applied to a dataset, by a judge whose agreement with your people is itself a number someone has to keep checking.

Which plan fits and why

One feature at $1,900 a month carries the part of this that is ongoing rather than one-off. Recalibration is not a task you finish, and one of our six guarantees is fixed on it: judge recalibrated against human verdicts at least monthly. The agreement statistics appear in the monthly drift (a change in how a feature behaves from one run to the next, measured against your dataset) report rather than being quoted once and forgotten. The plan also samples 1,000 production requests a month, which is how cases users complain about reach the dataset instead of only reaching support.

The honest limit: recalibration runs against labels your domain expert gives us, so if that person is unavailable the recalibration waits. An agreement figure describes the calibration set it was measured on and is not a statement about your whole production traffic. We do not replace human review, and we publish no measurement of our own on any model. If the reason you are here is that a customer asked for this in writing, read an enterprise customer asked for evaluation evidence, and the full bounds are on what we cannot do. The dataset and the first calibration come from the eval baseline, and every external figure on this page is listed on the evidence page.