What is actually happening
A prompt edit is a behaviour change with the same model on both sides of it. System text, few-shot examples, output schema, formatting instructions and tool descriptions are all prompt surface, and an edit to any of them can move what the feature produces on inputs nobody was looking at. There is no provider announcement to react to here, and the change log will not contain your incident.
The two-week gap is a reporting delay, not an incubation period. The edit changed behaviour the moment it shipped. What took two weeks was an input shape arriving that the edit had broken, or a complaint reaching someone who could connect it to a change that was by then buried in the history. By the time anyone looks, the edit is one of nine, and the obvious suspect is the most recent one rather than the responsible one.
Underneath that sits a coverage gap. Your golden dataset (a fixed set of labelled inputs and expected outputs, versioned, that every run scores against) did not contain the case the edit broke, which is unsurprising given where datasets start. DeepEval's documentation says even 10 to 20 inputs[src] is a great start.[Confident AI] Hamel Husain starts with around 30 examples[src] and keeps going until he sees no new failure modes.[Hamel Husain] Arize tells practitioners to begin by pulling 50 real traces[src] from their system.[Arize AI] Those are starting points, so an early dataset misses things by construction, and it only stops missing them if failures are fed back into it.
The second gap is that nothing scored the edit before it merged. Braintrust recommends running evals on every pull request to catch regressions (a case that passed before the change and fails after it) before they reach production.[Braintrust] Promptfoo documents running evaluations in CI/CD to test prompt changes before they reach production and to fail the build when quality thresholds (the number you and we write down as the line a score must not cross) are not met.[Promptfoo] Both are describing the check your edit did not pass through.
What to do this week
Finding the edit comes first because everything after it is guesswork otherwise. A list of dated edits next to a date the reports started is cheap to produce and narrows nine candidates to two or three.
Collecting the real failing inputs comes before any dataset work, because a regression you cannot reproduce is an opinion. Day three turns those inputs into labelled dataset items and versions the dataset, which is the step that makes the rest repeatable. Galtea writes that a golden dataset drifts as product scope changes and failure modes are discovered, that additions should require code review, and warns against updating ground truth to match model behaviour.[Galtea] That warning matters most on a day like day three, when the quickest way to make a test pass is to relabel it.
The rerun on day four compares two prompt versions against one pinned dataset, so the only variable is the edit. The gate comes last for a reason that is easy to miss: a gate over a dataset that lacks the broken case would have passed this edit too. Dataset first, gate second. Galtea describes a tiered gate in which blocking failures such as safety or policy violations block a deploy at any rate, while regressions gate on the change in fail rate, illustrated with a faithfulness fail rate rising from 3 percent to 5 percent[src].[Galtea] That figure is Galtea's example, not a recommended threshold. The number your build fails on is the one you agree with us and write down. Our threshold and gating protocol and dataset protocol set out both halves.
What happens without evals
The clearest published case is Anthropic's own. In a postmortem published on 23 April 2026, Anthropic attributed six weeks of Claude Code quality reports to three product-layer changes[src] made between 4 March and 20 April 2026, and said that neither its internal usage nor its evals initially reproduced the issues.[Anthropic] Anthropic states that the API was not impacted and that model weights were unchanged. One of those changes, a verbosity system prompt, was measured by Anthropic at a 3 percent quality drop[src] on Anthropic's own eval suite.
We make no claim that we would have caught it. The case is here because it is the best sourced demonstration of the mechanism you are looking at: product-layer and prompt-layer changes degraded a feature for weeks with no model change at all, reports accumulated, and the existing evals did not reproduce what users were describing. That is the shape of your incident, and the two fixes it points to are the same two: cases in the dataset, and a check before the merge.
Which plan fits and why
Up to eight features is $9,800 a month and is the plan that puts a gate in your pipeline, so an edit is scored before it merges rather than after someone complains. It covers eight features, 5,000 dataset items across them, 40 scheduled runs a month, and a flagged change comes back within 1 business day. A prompt edit is not the only kind of change that reaches your output this way. A change to retrieval (the step that fetches documents or records and puts them into the prompt) does the same thing, which is why both are named in the same commitment. The guarantee that applies here is fixed: every flagged prompt or retrieval change is evaluated before it ships, within the plan's turnaround. A retrieval change counts the same way an edit to system text does.
The honest limit: that price is for a platform team with several features and a pipeline to gate. If you have one feature and can flag changes by hand, One feature at $1,900 a month evaluates each flagged change before it ships within 2 business days, without the gate inside your pipeline. Either way a scorer (a function that turns one output into a number or a pass and fail verdict) only sees what the dataset contains, the gate only holds the threshold values you agreed, and neither is a statement about inputs nobody has collected yet. Start at the eval baseline, read how it works, and read the bounds on what we cannot do.