Running an eval suite in CI: what a gate does

A gate in continuous integration scores a change before it merges and fails the build when an agreed number is crossed. It changes the week in three ways: the cost of a prompt edit goes up slightly, the cost of a bad prompt edit goes down a lot, and arguments about quality move from after the deploy to before it.

What two vendors document

Braintrust recommends running evals on every pull request to catch regressions (a case that passed on the previous run and fails on this one) before they reach production. [Braintrust]

Promptfoo documents running evaluations in continuous integration to test prompt changes before they reach production and to fail the build when quality thresholds (the number a run has to stay inside before a change is allowed to ship) are not met. [Promptfoo]

Two vendors, same mechanism, described in almost the same words. That convergence is the reason to treat the practice as settled rather than novel. One caution when reading the promptfoo page: it contains a pass rate inside an example snippet, which illustrates how the check is configured and is not guidance on what your number should be.

What running on every pull request feels like

The cadence is the substantive choice, not the tool. Per pull request means every change to a prompt, a retrieval configuration or a pinned model version gets scored against the same cases before anyone merges it.

Three things change in practice.

Review gets a fact in it. A reviewer looking at a system prompt edit currently has an opinion. With a gate, they have a diff plus a list of the cases that changed verdict, which is a different conversation.

Small edits stop being free. Rewording one line of a prompt now costs a run. That is a real tax on the way many teams work, and it is the honest cost of the practice.

Blame moves earlier. When a regression is caught at the pull request, nobody spends Thursday afternoon working out which of six merges caused it.

Making it block rarely and correctly

A gate that blocks every second build gets disabled within a fortnight, and a disabled gate is worse than none because it looks like coverage.

Galtea recommends a tiered release gate in which blocking failures such as safety or policy violations trigger a deploy block at any rate, while regressions gate on the change in fail rate, giving the example of a faithfulness fail rate rising from 3 percent to 5 percent[src]. [Galtea] Those figures are Galtea's illustrative example in Galtea's own documentation, not a recommended threshold.

The shape that transfers is two tiers. A short list of failures that block at any rate, and everything else read as a movement against the previous run with a band that is reported rather than blocked. How those numbers get agreed is in setting eval thresholds that gate a release and in the EV-THRESH protocol.

Pin the dataset or the gate compares nothing

A per pull request gate compares two runs. If the cases moved between them, the comparison is meaningless and the failures it reports are noise.

Braintrust says every change to a dataset is tracked so experiments can pin to specific versions. [Braintrust] Weights and Biases documents an evaluation as a dataset of test examples plus one or more scoring functions, used to measure performance against repeatable test cases and identify regressions. [Weights & Biases]

So the run record for a gate carries the golden dataset (the fixed set of saved inputs and expected results the run is scored against) version, the scorer set and the model identifier. Refreshing the dataset in the same week you tighten a threshold is the classic way to spend two days debugging a comparison that was never valid.

Where a judge sets the score, the gate inherits its error

If the number your build fails on comes from a judge (a model that scores another model's output against written rules), then the reliability of the gate is the reliability of that judge.

Galtea recommends measuring a judge with accuracy, precision, recall, F1, Pearson correlation, Spearman correlation and Cohen's kappa rather than one statistic, and writes that a judge with 0.9 accuracy and 0.1 Cohen's kappa[src] is essentially guessing. [Galtea]

A gate resting on a judge should therefore name which statistics it reads and carry a dated calibration record for the judge behind it. Otherwise the first time it blocks a change everyone believes is fine, nobody can say whether the change was bad or the instrument was.

Record every override with a name, a date and a reason. Overrides that leave no trace turn a gate into a formality, and the trail is what tells you six months later whether the number was set too tight.

The class of change a gate cannot see

This is the limit worth understanding before a gate becomes the whole strategy.

A pipeline gate sees changes that travel through the pipeline. A provider side change does not. Nobody opens a pull request when a hosted model is updated, and Future AGI writes that OpenAI, Anthropic, Google and most frontier providers update model weights without a major version bump, which is that vendor's characterisation of the market rather than a provider statement. [Future AGI]

So a gate needs a second trigger alongside it: a scheduled rerun, and a rerun when a provider announces something. Those triggers are EV-RERUN, and what the providers have actually announced is tracked in the model change log.

A gate also only sees the cases in its dataset. A regression on traffic nobody wrote a case for passes silently, which is why production sampling runs beside the gate rather than after it.

Where to start

Start with one feature, a small pinned dataset and a single tier one rule that everyone agrees should block. Run it in advisory mode for a fortnight, look at what it would have blocked, then turn it on.

Wiring a gate into a customer pipeline is part of the eight feature plan. If you are here because an edit shipped and made things worse, the recovery sequence is on prompt edit regressed the output. And if the question underneath is whether to run this yourself at all, that is eval tool or eval service.