Summarisation is the feature type where teams most often skip evals, because the obvious test does not work. There is no single correct summary, so there is nothing to match a new output against. That is a reason to score differently, not a reason to score nothing.
What typically breaks in summarisation after a model change
Five patterns, and one of them is a process failure rather than a model failure. Each shows up as drift (a measured change in a feature's scores between two runs of the same dataset).
Coverage quietly narrows
The summary stays fluent, stays the right length, and drops the one paragraph your users needed. A new model version weighs salience differently, so the point that used to survive compression now does not. Nothing looks wrong. The support ticket arrives two weeks later from someone who acted on a summary that left out the exception clause.
This is the failure that a golden dataset (a fixed, versioned set of inputs with the verdict a domain expert gave each one) catches only if the required points were written down per document at build time.
Claims appear that are not in the source
The opposite failure. A detail, a number or a causal link shows up in the summary that the source document does not support. Fluency makes it hard to spot by reading, because an invented sentence reads exactly like a supported one. The check has to be per claim rather than per summary. The equivalent measurement for a feature that answers from documents you fetch is defined formally on the RAG hub.
Length, structure and verbosity drift
Summaries that were four bullets become nine. A section header appears that your interface does not render. This is the cheapest failure to detect and the most common after any change.
It also happens without a model swap. In a postmortem published on 23 April 2026, Anthropic attributed six weeks of Claude Code quality reports to three product-layer changes and measured a 3 percent quality drop[src] from one system prompt change on its own eval suite. [Anthropic] Anthropic states that the API was not impacted and model weights were unchanged. The useful reading for you is that a verbosity prompt is a change worth running.
The reference gets rewritten to match the new model
This one is a process failure rather than a model failure, and it is the most damaging. The model changes, the stored expected summaries stop matching, and someone updates the expected summaries. The suite goes green and measures nothing.
Galtea writes that a golden dataset drifts as product scope changes and failure modes are discovered, that additions should require code review and deletions should be treated as production risk, and warns against updating ground truth to match model behaviour. [Galtea] Braintrust says every change to a dataset is tracked so that experiments can pin to specific versions, which is the tooling side of the same rule. [Braintrust]
The judge moves under the rubric
Coverage and invention are both judged, so both inherit the judge (a model that scores another model's output against a written standard)'s own movement. Future AGI writes that a judge model can move 3-5 points[src] on a fixed rubric (the written scoring standard a judge or a human labeller applies to an output) in a single month with no change on your side. [Future AGI]
Which scorers apply
LangSmith documents four evaluator types: human review, rule based code functions, LLM as judge in reference free and reference based variants, and pairwise comparison of two application versions. [LangChain] Summarisation leans on the reference free judge, because a reference based scorer (one automated check that turns an output into a number or a pass and fail verdict) needs an expected result and there is no single expected summary to store. Each scorer is recorded with its limit.
- Coverage judge scorer. Catches a required point that went missing, when the point is on your expert's per-document list. Misses anything nobody listed, and says nothing about whether the summary reads well.
- Unsupported claim judge scorer. Catches statements the source does not support. Misses claims that are supported but irrelevant, and misses errors that came from a wrong source document.
- Length and format code check. Catches bullet counts, section headers, word ceilings, markdown your interface cannot render. Misses everything about meaning. It is the cheapest scorer here and it belongs in every set.
- Required element code check. Catches a missing date, a missing case number, a missing disclaimer, anything with a fixed shape. Misses paraphrase, so it is a floor and not a coverage measure.
- Pairwise comparison. Catches which of two prompt versions your expert prefers, useful when no absolute scale exists yet. Produces no standing number to track month to month.
Because everything interesting here is judged, calibration is not optional. Galtea writes that a judge must be run against a labelled gold set, with the gap between its verdicts and the human labels measured and the prompt iterated until the gap closes. [Galtea] Hamel Husain recommends about 100 examples per failure mode[src] to validate a judge, and writes that below 60 examples the confidence intervals are often too wide to support a useful conclusion. [Hamel Husain] The procedure is EV-JUDGE; scorer selection is EV-SCORE.
What a baseline includes for this feature type
The eval baseline for a summariser is built around source documents, not around expected outputs.
- Source documents sampled from your real inputs, spread across the lengths and types you actually receive, including the long and messy ones teams leave out.
- A per-document list of required points, written by your domain expert, which becomes the coverage rubric rather than a target summary.
- A claim-level rule for what counts as supported by the source, agreed in writing before any score exists.
- Format constraints expressed as code checks: length ceiling, structure, required elements, anything your interface depends on.
- A calibrated judge for coverage and invention, with more than one agreement statistic reported and a monthly recalibration cycle.
- A written rule for changing the dataset, with a refresh cadence. Future AGI recommends refreshing a golden dataset monthly by default, sampling 200-500 recent production traces[src] stratified by intent and persona, weekly during rapid change and immediately after model or corpus updates. [Future AGI]
Which plan fits
One feature at $1,900 a month fits a single summariser, after the baseline. It covers reruns on every tracked provider update from the change log, on every prompt change you flag, and weekly, with a monthly drift report. The caveat for this feature type is the honest one: coverage and invention are judged, so your numbers are only as good as the judge behind them, and a judge is a model behind an API like any other. That is why judge recalibration is on a monthly cycle rather than left until a number looks wrong. Every figure on this page traces to a source in the evidence list.