A model update changed our outputs

What is actually happening

Two different things get called a model update, and they need different responses. The first is your code calling a moving name. Google says a Gemini "latest" alias points to the most recent release of a model variation and is hot-swapped with every new release of that variation, so what your feature calls can change with no deploy on your side.[Google] Google's own changelog records the latest aliases being pointed at Gemini 3 models on 21 January 2026, and OpenAI's changelog records the chat-latest snapshot being updated on 24 June 2026 to point to the latest Instant model in ChatGPT. Both sit in our change log with the provider's own source attached.

The second is a change to the call rather than to the model. Anthropic deprecated the temperature, top_p and top_k request parameters from Claude Opus 4.7 onward, and a non-default value now returns a 400 error.[Anthropic] That breaks caller code with no model swap by you, and it says nothing about output quality. Reading which of the two you are looking at is most of the work, and the per-provider change log is where that reading starts.

Separately, Future AGI writes that OpenAI, Anthropic, Google and most frontier providers update model weights without a major version bump.[Future AGI] That is Future AGI's characterisation of the market and not a provider statement, so we label it that way every time we use it. Google's own documentation is the firmer ground: it says stable models usually don't change and that most production apps should use a specific stable model.[Google]

The honest part is what you have right now. You have an impression that outputs moved. Without a golden dataset (a fixed set of labelled inputs and expected outputs, versioned, that every run scores against) you cannot say how much moved, on which inputs, or whether anything moved at all rather than your reading of it changing. That gap is what the week below closes.

What to do this week

The sequence in the list above has one rule behind it: stop the movement before you try to measure it. re-pin (moving your code from a floating model alias to a fixed, dated model id) on day one is not the fix, it is the freeze. It costs a deploy, it is easy to revert, and it means that everything you measure from day three onward is measuring one configuration instead of a moving target.

Reading the provider's own page comes second because it narrows the search. A parameter deprecation, a snapshot shutdown and an alias repoint produce different symptoms and need different fixes, and the provider has usually written down which one it shipped. Guessing here costs days.

The rerun comes third, and it runs against both model ids rather than against an absolute bar, because the only question you can actually answer is what changed between the two. Day four is manual on purpose. A scorer (a function that turns one output into a number or a pass and fail verdict) tells you that eleven cases flipped; only reading them tells you whether those eleven matter. The decision waits until last because a decision made before day four is a guess with a deployment attached. Our rerun protocol is the same sequence written out step by step.

What happens without evals

The published case we can point to is a provider's own, and it is a lesson about attribution rather than a sales point. In a postmortem published on 17 September 2025, Anthropic wrote that its evaluations simply didn't capture the degradation users were reporting, and that it had relied too heavily on noisy evaluations.[Anthropic] Anthropic's stated causes were three overlapping infrastructure bugs[src], a context window routing error, output corruption and a TPU miscompilation, and not a change in model weights.

We make no claim that we would have caught it. We were not measuring it, and the stated causes sat inside the provider's own serving stack. What the case does show is narrower and still worth having: people experienced worse outputs and reached for the nearest available explanation, a model update, and the explanation the provider published was somewhere else entirely. Without a measurement on your own inputs, you cannot tell which explanation applies to you, and you will spend the week rewriting a prompt that was never the problem.

Which plan fits and why

One feature is $1,900 a month and is built for exactly this: one production feature that a provider update can move. It reruns your suite on every tracked provider update, on every prompt or retrieval change you flag, and weekly, samples production, and sends a monthly drift (a change in how a feature behaves from one run to the next, measured against your dataset) report. One of our six guarantees covers the trigger you are living through now: every tracked provider update triggers a rerun within 2 business days of the provider's announcement.

The honest limit: the plan covers one feature, 500 dataset items, 4 scheduled runs a month and a 1,000 request production sample, and a rerun measures the agreed dataset and the agreed threshold (the number you and we write down as the line a score must not cross) values. It says nothing about inputs your dataset does not contain, and a regression (a case that passed on the previous run and fails on this one) outside that dataset will not appear in the verdict. You need a dataset before any of this runs, which is the eval baseline, and the full bounds are written on what we cannot do.