Rerun triggers and procedure

Scope

This protocol covers when we run your evals again and how that run is carried out. Four events start a rerun on every plan: a tracked provider update, a flagged prompt change, a flagged retrieval (the step that fetches documents or records for the model to answer from) change, and the weekly schedule. The platform plan adds a fifth, a gate in your pipeline. Every rerun scores your golden dataset (the fixed set of labelled inputs and expected outputs we agreed at baseline), built in EV-DATA, with the scorers agreed for that feature, so runs can be compared over time and drift (a change in how a feature behaves from one run to the next, measured against that dataset) has something to be measured against.

This protocol does not set your thresholds. Those are agreed with you at baseline and are covered by EV-THRESH.

Turnarounds below are commitments about when we run and when you get the verdict. They are not statements about what a run will find.

Inputs

• The current golden dataset for the feature, versioned, with the last run on record.
• The scorers and the judge rubric agreed for that feature type.
• The thresholds agreed with you.
• The list of models, aliases and platforms the feature calls in production.
• The trigger signal itself, with its date and its source.

Steps

1. Watch the provider surfaces you depend on

We read the deprecation and changelog pages of each provider your tracked features call, and we re-read them rather than relying on a date noted earlier. Anthropic publishes a four state lifecycle for its models, active, legacy, deprecated and retired, and says requests to a retired model will fail.[Anthropic] Published dates themselves move: Google announced on 2 April 2026 that the Vertex AI retirement dates for Gemini 2.5 Pro, Gemini 2.5 Flash and Gemini 2.5 Flash-Lite had been updated to 16 October 2026[src].[Google]

2. Read the notice period the provider itself published

Three providers publish three different commitments. They are separate promises by three companies, not an industry norm, and we treat each feature by the platform it actually runs on. Anthropic says customers with active deployments get at least 60 days[src] before a publicly released model is retired, and its page states that partner platforms set their own schedules.[Anthropic] OpenAI says at least 6 months[src] for a generally available model, at least 3 months for specialized variants, and that preview models may be retired with much shorter notice, such as 2 weeks, with no minimum committed.[OpenAI] Google says Gemini preview models may be used in production but will be deprecated with at least 2 weeks[src] notice, which is why a feature pinned to a preview model gets a shorter rerun cycle than one on a general release.[Google]

3. Classify the change and choose a recommended action

Every entry gets one of four actions: rerun, re-pin (move your code from a floating model alias to a fixed, dated model id), re-prompt, or migrate. Two published facts drive that choice. Google says a Gemini "latest" alias points to the most recent release of a model variation and is hot-swapped with every new release of that variation, so what your code calls can change with no action on your side.[Google] Google's Gemini documentation also says stable models usually don't change and that most production apps should use a specific stable model, which is the ground for a re-pin recommendation.[Google] Separately, Future AGI writes that OpenAI, Anthropic, Google and most frontier providers update model weights without a major version bump; that is Future AGI's characterisation of the market, not a provider statement, and we label it that way wherever we use it.[Future AGI]

4. Rerun on a tracked provider update

Every tracked provider update triggers a rerun within 2 business days of the provider's announcement, on every plan. A tracked update is not only a new model. OpenAI announced on 11 June 2026 that the gpt-5-2025-08-07 snapshot would be shut down on 11 December 2026[src], which shows that pinning to a dated snapshot buys time and not permanence.[OpenAI] Anthropic deprecated the temperature, top_p and top_k request parameters from Claude Opus 4.7 onward, and a non-default value now returns a 400 error, a change that breaks caller code with no model swap on your side.[Anthropic] Both kinds start a run.

5. Rerun on a flagged prompt or retrieval change

You flag a change before it ships, and it is evaluated before it merges. Braintrust recommends running evals on every pull request to catch regressions (a case that passed on the previous run and fails on this one) before they reach production, which is the same idea moved out of your pipeline and into ours on the plans without a CI gate.[Braintrust] A prompt change is any edit to system text, few-shot examples or output schema. A retrieval change is any edit to the corpus, chunking, embedding model, index or ranking. We evaluate the flagged change against the same dataset and thresholds as the last passing run, and return a verdict within 2 business days, or within 1 business day on the platform plan. The comparison is the point: a flagged change is read against the previous run, not against an absolute score.

6. Run the weekly schedule

A scheduled run goes ahead whether or not anything was announced. Google's own documentation is the reason: a "latest" alias is hot-swapped with every new release of that model variation, so a feature can be calling something different with no announcement to react to.[Google] Scheduled runs use the same dataset version and scorers as the last triggered run, so the two are comparable. Your plan sets how many scheduled runs a month are included: 4 on One feature, 12 on Up to three features, 40 on Up to eight features.

7. Gate the pipeline, on the platform plan

On the platform plan we wire the eval into your pipeline so a change is scored before it merges. Braintrust recommends running evals on every pull request to catch regressions before they reach production.[Braintrust] Promptfoo documents running evaluations in CI/CD to test prompt changes before they reach production and to fail the build when quality thresholds are not met.[Promptfoo] We follow that shape. The number the build fails on is the threshold you agreed with us, not a figure taken from either tool's example snippets.

8. Pin the dataset version and record the run

Before the run starts we pin the dataset version, so the comparison is between two runs of the same data and not two different datasets. Braintrust tracks every change to a dataset so that experiments can pin to specific versions, which is the behaviour we rely on where that tool is in use.[Braintrust] We record the trigger, the dataset version, the model id and platform, the scorer set, the rubric version and the run timestamp.

9. Write the verdict and hand it back

We write one verdict per run: what triggered it, which scores moved and by how much against the previous run, which cases changed verdict, whether anything crossed an agreed threshold, and the recommended action. A regression is listed by case, with the input and both outputs, so you can read the change yourself. When a provider announcement started the run, the change-log entry is published with the provider's own source, and the verdict feeds the monthly drift report.

Outputs

• A rerun on a pinned dataset version, with trigger, models and versions recorded.
• A written verdict naming every score that moved and every case that changed verdict.
• A recommended action: rerun, re-pin, re-prompt or migrate.
• A change-log entry with the provider source, when a provider announcement started the run.
• An input to the monthly drift report.

Limits

Version and derivation

Version 1.0, derived from provider changelog practices, Braintrust CI integration, written 22 September 2026, by DriftCaught.