Synthetic example

Sample drift report

This is the shape of what arrives by the 5th business day of each month. Every number on this page is invented. There is no customer here and no measurement of any model.

Synthetic. Feature names, model ids, scores and thresholds below are made up to show the structure of the document. Nothing on this page is a real result and nothing on it is a measurement of a model's behaviour.

Example Ltd, August 2026, Up to three features

Monthly drift report

Protocols run: EV-DATA 1.0, EV-SCORE 1.0, EV-JUDGE 1.0, EV-THRESH 1.0, EV-RERUN 1.0, EV-PROD 1.0, EV-REPORT 1.0. Datasets pinned at v14, v9 and v6.

Features covered: 3
Runs this month: 17
Scorers below threshold: 2
Judge agreement: 0.81

1. What was covered

Chat assistant

520 items, v14

Support answers in product. Dataset built from 2025 and 2026 escalations.

RAG answers

610 items, v9

Answers over the policy library. Dataset built from cases where the answer was unsupported.

Ticket extraction

370 items, v6

Fields pulled from inbound email. Dataset built from fields a human had to correct.

Coverage comes before any score, because a score you read without knowing its dataset is a number without a subject. See EV-REPORT.

2. Runs and what triggered them

Runs in the reporting month, with triggers and verdicts
Date	Trigger	Feature	Verdict
2026-08-03	Weekly schedule	All three	No change above threshold
2026-08-07	Provider update, tracked model	Chat assistant, RAG answers	Refusal correctness below threshold on chat assistant
2026-08-11	Flagged prompt change	Chat assistant	Passed, shipped
2026-08-18	Flagged retrieval change, new chunker	RAG answers	Citation present below threshold, held
2026-08-24	Provider update, pricing and context	All three	No change above threshold

3. What moved, per scorer

The report shows the change against the previous run and the threshold you agreed, not just the current value. A number that has not moved is as much of a result as one that has.

Chat assistant, after the provider update of 2026-08-07
Scorer	Before	After	Delta	Threshold	Verdict
Rule following, code check	0.97	0.97	0.00	0.95	Pass
Refusal correctness, judge	0.91	0.86	-0.05	0.88	Below threshold
Tone match, judge	0.88	0.84	-0.04	0.80	Watch
Answer length in range, code check	0.99	0.99	0.00	0.95	Pass

RAG answers, after the retrieval change of 2026-08-18
Scorer	Before	After	Delta	Threshold	Verdict
Retrieval hit rate, code check	0.84	0.85	+0.01	0.80	Pass
Faithfulness, judge	0.90	0.89	-0.01	0.85	Pass
Citation present, code check	1.00	0.93	-0.07	0.98	Below threshold
Answer relevance, judge	0.87	0.87	0.00	0.82	Pass

4. The judge, this month

Human verdicts collected: 60
Agreement with the expert: 0.81
Cohen's kappa: 0.62
True positive rate: 0.79
True negative rate: 0.84
Recalibrated: 2026-08-14

More than one statistic, because a judge can agree with your expert most of the time and still be useless on the cases that matter. See EV-JUDGE.

5. What production sent

Requests sampled: 4,812
Inputs unlike any dataset item: 11%
New failure modes proposed: 3
Accepted into the dataset: 2

This is how the dataset stays honest. If your users start asking things the dataset has no case for, the suite keeps scoring well while the feature gets worse. See EV-PROD.

6. What the dataset does not cover

Chat assistant: no cases in any language other than English, though 4% of sampled traffic was not English.
RAG answers: no cases where the policy library disagrees with itself. We know these exist and have no graded examples.
Ticket extraction: no cases from the two customers onboarded in July.
None of the three has a case for the failure mode raised in your 12 August escalation. It is proposed for v15.

This section is mandatory in every report. A score is a statement about the dataset it was measured on, and the honest way to say that is to say what is missing.

7. Recommended actions

Chat assistant: refusal correctness is below the threshold you agreed. Twelve items moved, nine of them refusals that are now answered. Review the nine and decide whether the threshold or the prompt is wrong.
RAG answers: citation presence fell after the chunker change. The generation is unchanged; the citation span is being cut. Hold the chunker change.
All three: add the 12 August escalation to the dataset for v15.
Ticket extraction: nothing moved. Next scheduled run 2026-09-07.

The report says this about itself

Every score here is a statement about the dataset it was measured on, at the version named. It is not a statement about the feature in general, and it is not a prediction about the next request your users send. Coverage is bounded by the dataset and the plan.

Questions about the report

Is this a real customer's report?

No. Every number, feature name and model on this page is invented to show the structure. We do not publish customer results, and we publish no measurement of any model.

When does the real one arrive?

By the 5th business day of each month. That is one of the six written guarantees on every plan.

Why does it say what the dataset does not cover?

Because a score is a statement about the dataset it was measured on. A report that shows only the score invites you to read it as a statement about your feature, which it is not. EV-REPORT makes that section mandatory.

What happens when a scorer comes back below threshold?

You get the verdict, the items that moved, and a recommended action. We do not ship or hold your release. That decision is yours, and the threshold was yours to agree in the first place.

Can we get it in our own template?

On the agency partner plan, yes. Reports are white-label and the method is the published one, named by protocol version.

Start a plan Read EV-REPORT