What a golden dataset stops covering

A golden dataset (a fixed set of saved inputs with the expected result for each one, which an eval run scores against) stops covering your feature long before anything in it starts failing. The usual cause is not a model release. It is your own users, asking things the set was never built from. It is a snapshot of the traffic you had on the day you built it, and traffic does not hold still.

A golden freezes the input side

DeepEval defines a golden as a precursor to a test case that needs only an input plus expected results, and that becomes a full test case once the application generates the actual output at evaluation time.[Confident AI] That definition is DeepEval's, and other tools use the word differently, but it makes the limit easy to see. The input side is frozen at the moment someone saved it. Everything else keeps moving.

Galtea writes that a golden dataset drifts as product scope changes and failure modes are discovered, that additions should require code review, and that deletions should be treated as production risk.[Galtea] Those three sentences describe a dataset as a piece of production infrastructure rather than a file written once at baseline.

Four ways production walks away from the set

New intents. You shipped a feature. People now ask the assistant to do something the set has no case for. Nothing in the suite represents it, so nothing in the suite can fail on it.

New phrasing. The same intent arrives worded differently, because the UI changed, because a help article told people what to type, or because a different team started using the tool.

New material on the retrieval (the step that fetches documents from your corpus and puts them in the prompt before the model answers) side. The corpus gains a document type, a template changes, a legacy export gets loaded. The questions look identical and the context behind them does not.

New users. A segment you did not have at baseline now makes up a real share of requests, with its own vocabulary and its own idea of a correct answer.

None of these produce a failing run. They produce a set that answers a question you stopped asking.

Why the score can hold while the feature gets worse

Two different things are being confused when a green suite is read as good news.

A regression (a case that passed on the previous run and fails on this one) is loud. A scorer that passed now fails, the run shows a delta, someone gets a verdict. That machinery works.

A coverage gap is silent by construction. No scorer fires, because no case exists to fire on. The suite keeps scoring the traffic you had in March at the same rate it always did, while the traffic you have in September is served by a feature nobody is measuring. The number on the report is accurate and it is answering the wrong question.

The check is production, not the suite. Langfuse says observation level evaluators are the recommended target for live production data, and lists sampling a percentage of traces as a way to manage the cost of judge based evaluation.[Langfuse] Langfuse does not publish a recommended percentage, so we do not quote one. Our production sampling protocol scores a sample of live traffic against the same rubric (the written rules a judge scores against, agreed with you at baseline) the suite uses, which is what makes the two comparable. Items that the sample turns up and the set has no equivalent for become candidates for the next refresh.

What a refresh cadence looks like

Future AGI recommends refreshing a golden dataset monthly by default, sampling 200 to 500[src] recent production traces stratified by intent and persona, labelling them against the active rubric, and versioning the dataset as a new artifact.[Future AGI] The same page gives weekly during rapid change, and a refresh immediately after a model or corpus update.

Three parts of that are worth separating, because only one is a number.

The stratification is the part doing the work. A sample drawn evenly across intents and personas keeps a high volume intent from crowding out the rare ones, which are usually where the interesting failures sit.

The versioning is what makes a refresh safe to do. Braintrust says every change to a dataset is tracked so that experiments can pin to specific versions.[Braintrust] That is a description of what the tool provides, and it is the property you need from whatever you use: a run that names the dataset version it scored, so a drop in score can be read as a change in the feature rather than a change in the set.

The cadence is the part to argue about. Monthly is Future AGI's default, not a law. A feature whose scope has not moved in a quarter does not need a monthly relabel. A feature shipping weekly does.

The rule that keeps a refresh honest

Galtea warns against updating ground truth to match model behaviour.[Galtea] This is the failure mode that turns a refresh into a way of making a problem disappear. A case fails, someone looks at the output, the output is plausible, and the expected result gets edited to accept it. Do that a few times and the dataset records what the model does rather than what your business needs.

Our dataset protocol treats an added case, a removed case and an edited expected result as three separate events, each with a reason written in the dataset changelog. An edit that says "model produced this instead" is not a reason. An edit that says "the original label was wrong, here is why" is.

The report has to state what the set does not cover

A refresh cadence handles what you know changed. The coverage statement handles what you do not.

Every monthly report we send carries a section naming what the dataset does not cover, and a bounds statement placing each number on the dataset version it was scored against. Not as a disclaimer at the bottom, as a section someone has to read. The sample report shows both in place. We cannot make a model deterministic or guarantee its outputs, and coverage is bounded by the dataset and the plan, which is written out in full on what we cannot do.

If you are still choosing a starting size for a first set, the published figures and how they differ are in how many examples a golden dataset starts from. If your feature reads from a corpus, the scorers and the specific coverage traps are on the retrieval feature hub. What a baseline includes, and what it does not, is on the baseline page.