Golden dataset construction

Scope

This protocol covers how we build and maintain a golden dataset (a fixed set of saved inputs with the expected result for each one, which an eval run scores against) for one feature. It starts from real traffic, ends with a versioned file and a refresh schedule, and is the input to every other protocol we run for that feature.

It does not choose the scorers (the functions that turn one output into a score or a pass and fail verdict) that run against the dataset. That is EV-SCORE. It does not set the pass mark either. Thresholds are agreed with you in EV-THRESH, and this protocol only produces the set of cases those thresholds are read against.

Inputs

• Read access to production traces or logs for the feature in scope.
• A named domain expert on your side who can say whether an output is acceptable.
• A written description of a good output, in your words, including anything that is a policy or legal requirement.
• The failure modes you have already seen, from support tickets, internal review or user reports.
• A note of any data that must be redacted or excluded before it leaves your systems.

Steps

1. Fix the feature and name its failure modes

Write down one feature and one feature type. A dataset that mixes a support chat assistant with a document extraction job cannot be scored by one set of scorers, and a change that hurts one will be hidden by the other.

Then list the ways the feature is already known to fail. Hamel Husain's published heuristic is to start with around 30 examples[src] and keep adding until no new failure mode appears in the new examples, which makes the list of failure modes, not a target count, the thing that decides when the first pass is done. [Hamel Husain]

• Name each failure mode in one line, in plain language.
• Keep the list open. New modes get added as they are found.

2. Pull real traffic, not invented examples

Sample from what the feature actually received. Arize's guide on judge (a model that scores another model's output against a written standard) scorers tells practitioners to start by pulling 50 real traces[src] from their system. [Arize AI] Arize's page does not say those traces have to be failures, and we do not filter to failures at this stage, because a dataset of only failures cannot show a change that breaks something that used to work.

DeepEval's documentation puts the floor lower and says even 10 to 20 inputs[src] is a good start for an evaluation dataset, and that production traces can be used to curate the next one. [Confident AI] We treat that as a floor for getting something running on day one, not as the size of a baseline you would sign off.

The published starting figures differ because they answer different questions. We agree the first size with you against the failure mode list from step 1, and record which figure we started from.

3. Write each item as a golden, not as a finished test

Each row is stored as a golden (DeepEval's term for a precursor to a test case that holds an input plus the expected result, and becomes a full test case when your application produces the actual output at evaluation time). [Confident AI] Storing the input and the expected result separately from any output means the same row can be rerun against a new model version without being rewritten.

• Store the input exactly as the feature received it, after redaction.
• Store the expected result in the form the scorers will read, for example the expected fields for extraction or the expected supporting context for retrieval.
• Store the metadata you will want to slice by later: intent, customer segment, language, channel.

4. Have your domain expert label the set

Labels come from your side. Hamel Husain writes that in most organisations one or two key individuals hold the judgment that matters for the AI product, and that an automated judge is aligned iteratively against that person's own critiques. [Hamel Husain] We run the sessions, record the critiques and turn them into written label rules. We do not supply the expert and we do not overrule their labels.

• Label in short sittings, with the failure mode list open.
• Capture the reason in free text, not only the verdict. The reasons become the rubric (the written rules a scorer or a judge applies to decide whether an output passes).
• Where two reviewers disagree, record both readings and take the rule back to the expert.

5. Size the calibration subset per failure mode

Part of the dataset is reserved for checking a judge against human labels. Hamel Husain recommends about 100 examples per failure mode[src] to validate an automated judge, and writes that below 60 examples the confidence intervals are often too wide to support a useful conclusion. [Hamel Husain]

That figure sizes the calibration subset, not the whole dataset. The calibration procedure itself is EV-JUDGE. What this protocol does is make sure enough labelled rows exist per failure mode for that procedure to say anything.

6. Pair the dataset with its scorers

A dataset on its own produces no result. Weights and Biases documents a Weave Evaluation as a dataset of test examples plus one or more scoring functions, used to measure performance against a fixed set of test cases, compare changes over time and identify regressions (a case that passed on the previous run and fails on this one). [Weights & Biases] We follow the same shape: the dataset file names the scorer set it is meant to be run with, and EV-SCORE decides what goes in that set.

7. Version the dataset and pin every run to a version

Braintrust tracks every change to a dataset so that experiments can pin to a specific version. [Braintrust] We use that behaviour where you already run Braintrust, and reproduce the same rule in file form where you do not: every dataset is written as a new numbered version, never edited in place.

Inside the service, each run record stores the dataset version alongside the run id, the scorer set and the model identifier it was run against. A score with no dataset version attached is not reported.

8. Govern additions and deletions

Galtea writes that a golden dataset drifts as product scope changes and failure modes are discovered, that additions should require code review, that deletions should be treated as a production risk, and warns against updating ground truth to match model behaviour. [Galtea] That last warning is the rule we hold hardest. When a run fails and the model's new output looks reasonable, the expected result is not edited to match it unless your expert decides the old expectation was wrong, and that decision is recorded as a decision, not as a fix.

• Additions go through review by the same expert who labels.
• Deletions are recorded with a reason and kept in the changelog.
• A changed expected result is a new dataset version and is called out in the next monthly report.

9. Refresh on a schedule, and after a change

Future AGI recommends refreshing a golden dataset monthly by default, sampling 200 to 500 recent production traces[src] stratified by intent and persona, labelling them against the active rubric and versioning the result as a new artifact. The same page gives weekly during periods of rapid change, and immediately after a model or corpus update. [Future AGI]

We take monthly as the default cadence, agree the stratification with you, and pull the refresh forward when the change log records an update to a model you use, or when EV-RERUN flags a change to your retrieval corpus.

Outputs

• A versioned golden dataset file for the feature, with the failure mode of each row recorded.
• A labelled calibration subset, sized per failure mode, handed to EV-JUDGE.
• A dataset changelog: every addition, deletion and changed expectation with a reason and a date.
• A written label rule sheet, in your expert's words, that the rubric is built from.
• A refresh schedule naming the cadence, the sample size and the stratification.
• A named scorer set for the dataset, produced by EV-SCORE.

Limits

Version and derivation

Version 1.0, derived from Galtea evaluation guide 2026, Braintrust evaluation guide, Confident AI docs, written 22 September 2026, by DriftCaught.