Thresholds and release gating

Scope

This protocol covers the threshold (the number a run has to stay inside before a change is allowed to ship) for each scorer (the rule or model that turns one output into a score) on your feature, and the check that enforces it. It gives the method for agreeing those numbers with you. It does not set them for you, and it does not promise that a model will hold any score.

A threshold is an agreement about risk. Two companies running the same feature on the same model can reasonably agree different numbers, because the cost of a bad answer is different for each of them.

Inputs

• A pinned version of your golden dataset (the fixed set of test cases a feature is scored against), so the same examples are used before and after.
• The scorer list for the feature, with each scorer's range written down.
• A baseline run on the configuration that is live today.
• Your list of failures that block a release at any rate, such as safety or policy breaches.
• The place in your pipeline where a check can stop a change, and who can override it.

Steps

1. Fix what the number is measured on

A threshold means nothing until the thing it measures is fixed. Weights and Biases documents an evaluation as a dataset of test examples plus one or more scoring functions, used to measure performance against repeatable test cases, compare changes over time and identify regressions (a case that passed on the previous run and fails on this one). [Weights & Biases] We write the gate against that pair, built by EV-DATA and EV-SCORE, never against an ad hoc run.

2. Pin the dataset version behind the baseline

A number is only comparable to a number taken on the same examples. Braintrust tracks every change to a dataset so that experiments can pin to specific versions. [Braintrust] Whichever tool your stack uses, we record the dataset version with the baseline run, and a gate compares like with like or it does not run.

3. Separate blocking failures from regressions

Not every failure belongs on a sliding scale. Galtea recommends a tiered release gate in which blocking failures such as safety or policy violations trigger a deploy block at any rate, while a regression gates on the change in fail rate instead. [Galtea] We ask you which failures sit in the first tier before any number is discussed, because those are not traded off against a score.

4. Gate regressions on the change, not the absolute score

For the second tier the gate watches movement. Galtea's published illustration of that design is a faithfulness fail rate rising from 3 percent to 5 percent[src]. [Galtea] That is Galtea's example of how a delta gate reads, in Galtea's own documentation. It is not a threshold we recommend and not a number we carry into your plan.

What we do carry in is the shape: an agreed starting rate, an agreed size of move that blocks, and an agreed size of move that is reported but allowed through. The illustration is sourced on the evidence page.

5. Use more than one statistic where a judge sets the score

When a judge (a model that scores another model's output against written rules) produces the score, a single statistic can hide a broken instrument, which is why the calibration in EV-JUDGE matters before any gate is set. Galtea recommends measuring a judge with accuracy, precision, recall, F1, Pearson correlation, Spearman correlation and Cohen's kappa rather than one statistic, and writes that a judge with 0.9 accuracy and 0.1 Cohen's kappa[src] is essentially guessing. [Galtea] A gate that rests on a judge-produced score therefore names which statistics it reads, and the judge behind it is calibrated under EV-JUDGE.

6. Agree the numbers with you and write them down

We propose a starting set from your baseline run, your blocking list and the cost you attach to each failure. You change what you want changed. The agreed numbers go into your plan with the date, the dataset version they were set against and the name of the person who agreed them.

A threshold is an agreement, not a promise about the model. We can tell you that a run crossed a line you agreed, on the dataset you agreed, on a date. We cannot make a model hold a score. The full list of what we do not do is on what we cannot do.

7. Run the gate in the pipeline and let it fail the build

The gate has to sit where the change passes. Promptfoo documents running evaluations in continuous integration to test prompt changes before they reach production and to fail the build when quality thresholds are not met. [Promptfoo] The pass rate in their documentation is a figure inside an example snippet, so we read it as an illustration of the mechanism rather than as guidance on a number. Braintrust recommends running evals on every pull request to catch regressions before they reach production. [Braintrust] We follow that cadence: every change to a prompt, a retrieval configuration or a pinned model runs the gate.

8. Record every breach and change a threshold only by decision

When a run breaches, the check blocks and we send the failing examples, the scorer that moved, the size of the move and the dataset version. Nothing is re-scored to clear the block. A threshold changes only when you decide to change it, and the old number, the new number and the reason are kept in the plan's revision history.

Outputs

• One written threshold per scorer, with the dataset version and date it was agreed against.
• A separate blocking list that is not traded off against any score.
• A pipeline check that fails the build on a breach and names what moved.
• A dated breach log with the decision taken on each one.
• A revision history showing every change to a threshold and why it was made.

Limits

Version and derivation

Version 1.0, derived from FutureAGI regression testing, W&B evaluation practices, written 22 September 2026, by DriftCaught.