Reference-free scoring: what replaces the expected answer

When there is no reference answer, the taxonomy of evaluators splits in two, and only one half stays available. That is true whether the output is a summary, an open ended answer, or anything else a model writes freely, not just one feature type. What replaces the missing reference is a written rubric (the written rules applied to decide whether an output passes, by a model or by a person), scorers that read only the input and the output, and a hand labelled sample from the one person whose judgment counts.

The scorer type the problem rules out

Published evaluator taxonomies make the split explicit. LangSmith documents four evaluator types[src]: human review, rule based code functions, LLM as judge (a model that scores another model's output against written rules) in reference free and reference based variants, and pairwise comparison of two versions of an application. [LangChain] The reference free half of that list is the one that still works here, and it is what the rest of this post is about.

That reference free and reference based split is the whole article. A reference based scorer needs an expected result stored next to the input, in the shape DeepEval calls a golden (a precursor to a test case holding an input plus the expected result, which becomes a full test case once your application produces the actual output). [Confident AI]

For an invoice field, that expected result exists and is not controversial. For a 200 word summary of a support thread, twenty competent writers produce twenty different acceptable summaries, and picking one of them as the reference means punishing the other nineteen.

So the reference based half of the taxonomy is unavailable, and the rest of the work is deciding what fills the gap.

What a rubric replaces it with

The thing that survives the loss of a reference is the standard itself. A rubric says what an acceptable summary must do, in your own words:

• Every material commitment made in the thread appears.
• No figure appears that was not in the source.
• The customer's requested action is stated in the first sentence.
• Nothing is attributed to a person who did not say it.

None of those needs a canonical correct summary. All of them are checkable against the source document and the output. This is the move: from "does it match the answer" to "does it satisfy the conditions", which is a question you can still ask when no answer exists.

The rules come from someone specific. Hamel Husain writes that in most organisations one or two key individuals hold the judgment that matters for the AI product, and that the judge is aligned iteratively against that person's own critiques. [Hamel Husain] Their critiques of real outputs become the rubric lines. This is why the method starts with labelling rather than with metric selection, as set out in the EV-DATA protocol.

Sizing is the same problem as anywhere else. Hamel Husain's heuristic is to start with around 30 examples[src] and keep going until no new failure modes appear. [Hamel Husain] The three published starting points and what separates them are set out in how many examples a dataset starts from.

Grounding is measurable even when quality is not

For anything that summarises or answers from source material, one property is checkable without a reference answer.

Ragas defines faithfulness as the share of claims in a response that can be inferred from the retrieved context, computed by identifying every claim and checking each one against the context, expressed as supported claims divided by total claims[src] on a 0 to 1 scale. [Ragas]

A fabricated figure in a summary is a claim that cannot be inferred from the source, and that is exactly what this kind of scorer is built to find. Be precise about what you have bought, though. Faithfulness scores grounding in the material that was supplied. It does not score whether that material was true, and it does not score whether the summary was useful. See the summarisation feature page and the RAG page for how it sits in a wider set.

Grounding and usefulness are different properties. A summary can be perfectly faithful to its source and still omit the one commitment the customer cared about, which is why the rubric lines above are scored separately.

The judge still has to be checked

Once a judge is reading a rubric rather than comparing against a reference, it is carrying more of the verdict, not less. That raises the bar on calibration rather than lowering it.

Galtea writes that a judge must be run against a labelled gold set, with the gap between its verdicts and the human labels measured and the prompt iterated until the gap closes. [Galtea] The gold set here is labelled against the rubric, not against model summaries, and the same page recommends versioning judge prompts alongside application code.

The statistics to report and the floors below which no conclusion is published are covered in EV-JUDGE and in calibrating a judge against human verdicts.

The rule that gets broken most

There is one failure mode specific to features with no ground truth, and it is quiet.

Galtea writes that a golden dataset drifts as product scope changes and failure modes are discovered, that additions should require code review, that deletions should be treated as production risk, and warns against updating ground truth to match model behaviour. [Galtea]

When there is no canonical answer, the expected result is a human verdict, and a human verdict is easy to revise after the fact. A run fails, the new summary reads well, and the verdict quietly flips from fail to pass. Do that three times and the dataset now measures whatever the model currently does, which is a measurement of nothing.

The discipline is to treat a flipped verdict as a recorded decision with a name and a date on it, reviewed by the same expert who wrote the rubric, not as a correction.

Production traffic has no references at all

Scheduled runs on a fixed dataset only cover the cases in that dataset. Real traffic is where the untested inputs live, and none of it comes with an expected answer.

Langfuse says observation level evaluators are the recommended target for live production data, and lists sampling a percentage of traces as a way to manage the cost of judge based evaluation. [Langfuse] That page publishes no recommended percentage, so we do not quote one. The rate comes from your traffic and your plan allowance.

Reference free scorers are the ones that can run there, which is the practical reason to know which of your scorers is which. The sampling method is EV-PROD, and the ceiling on what any of it proves is on what we cannot do.