How many examples a golden dataset starts from

There is no single published number. Three widely read guides give three different starting sizes, because each is answering a slightly different question. Pick the one whose question matches yours, write down which you used, and move on.

The three published starting points

A golden dataset (a fixed set of saved inputs with the expected result for each one, which an eval run scores against) has to start somewhere, and the published advice does not agree on where.

DeepEval's documentation sets the floor lowest. It says that even 10 to 20 inputs[src] is a great start for an evaluation dataset, and that production traces can be used to curate the next one. [Confident AI]

Hamel Husain's heuristic sits in the middle. He writes that he starts with around 30 examples[src] and keeps going until he sees no new failure modes. [Hamel Husain] The number is the entry point, not the target. The stopping rule is the part that does the work.

Arize's guide to judge (a model that scores another model's output against written rules) scorers starts higher. It tells practitioners to begin by pulling 50 real traces[src] from their system. [Arize AI] Note the wording. Real traces, not real failures. The page does not say the 50 must be things that went wrong.

These are three separate published figures from three separate sources. They are not three estimates of one underlying number, and averaging them into a single recommendation would misrepresent all three.

Why the numbers differ

DeepEval is answering "what is the smallest thing that still counts as an evaluation". That is a question about getting something running on day one, and 10 to 20 is a fair answer to it.

Hamel Husain is answering "how do I find the failure modes". His heuristic is built around a stopping condition, so the count only tells you where to begin sampling. If example 31 shows you a mode you had not written down, you keep going.

Arize is answering "what do I need before I can build a judge scorer". That is a harder bar, because a judge has to be checked against human labels before its numbers mean anything, and checking needs material.

Three questions, three answers. The honest position is to say which question you are asking.

What actually decides your number

The failure mode list decides it. Before sampling, write down in plain language every way the feature is already known to go wrong: the support tickets, the internal complaints, the things a reviewer keeps correcting. Then sample real traffic until new examples stop adding to that list.

That means the first dataset for a narrow extraction job can be genuinely small, and the first dataset for an open ended assistant will not be. Same method, different result, and the difference is visible rather than buried in a rule of thumb.

Each row is stored as a golden (a precursor to a test case that holds an input plus the expected result, and becomes a full test case once your application produces the actual output), which is DeepEval's definition and is the form that lets the same row be rerun against a new model without being rewritten. [Confident AI]

The labels come from your side. Hamel Husain writes that in most organisations there are one or two key individuals whose judgment is crucial to the AI product, and that the judge is aligned iteratively against that person's own critiques. [Hamel Husain] Finding that person is usually harder than finding the examples.

The full method, step by step with a source on every step, is written up as the EV-DATA protocol.

The calibration subset is a separate number

People often collapse two questions into one. How big is the dataset, and how big is the part of it used to check the judge, are not the same question.

For validating an automated judge, Hamel Husain recommends about 100 examples per failure mode[src], and writes that below 60 examples the confidence intervals are often too wide to support a useful conclusion. [Hamel Husain]

That figure sizes the calibration subset per mode, not the whole file. If you have four failure modes and you want an agreement statistic you can act on for each, the arithmetic gets uncomfortable quickly. It is better to know that at the start than to discover it when the first agreement number turns out to mean nothing. How the calibration itself runs is covered in the EV-JUDGE protocol and in our post on calibrating a judge against human verdicts.

The first version is not the last version

Whatever size you start at, the set ages. Future AGI recommends refreshing a golden dataset monthly by default, sampling 200 to 500[src] recent production traces stratified by intent and persona, labelling them against the active rubric and versioning the result as a new artifact, with weekly refreshes during periods of rapid change. [Future AGI]

Two rules keep that refresh from quietly destroying the baseline.

• Version, never edit in place. Braintrust tracks every change to a dataset so experiments can pin to a specific version. [Braintrust] Whatever tool you use, a score with no dataset version attached cannot be compared to anything.
• Do not move the expected answer to match the model. Galtea writes that additions should require code review, that deletions should be treated as production risk, and warns against updating ground truth to match model behaviour. [Galtea] When a run fails and the new output looks reasonable, that is a decision for your expert to record, not a fix to apply.

A dataset built this way covers failure modes someone has already described. It says nothing about a mode nobody has met yet, which is why the limits are written down on what we cannot do.

If you want the first version built, labelled and versioned against your own traffic, that is the eval baseline, and the ongoing plans are on the plans page.