Drift report structure

Scope

This protocol describes the monthly report we send you: what is in it, in what order, and what it says about its own limits. The report covers drift (a change in measured quality over time on a fixed dataset, with no change on your side) across every feature in your plan.

The structure is ours. It is a company definition, not a published standard, so it is not comparable line by line with another vendor's report. A synthetic example is on the sample report page. The steps below say plainly which parts rest on published practice and which are simply our own document.

Delivery is one of the six written guarantees: a monthly drift report by the 5th business day. The guarantee covers delivery of the report, nothing more.

Inputs

• Every run in the period, with its trigger, its timestamp and its pinned dataset version.
• Per-scorer (one automated check that returns a number or a label for one output) results for each run and the threshold (the number agreed with you at which a measured change counts as a finding on this dataset) values agreed with you.
• This month's calibration sample, with the human verdicts beside the judge (a model prompted with a written scoring standard to score another model's output) verdicts.
• The production sample drawn in the period, and how each sampled item was scored.
• The coverage list: features in scope this month, features out of scope.
• Any change-log entry that triggered a rerun, with the provider's own source.

Steps

1. Open with the period, the plan and the pinned versions

The first page states the month, the plan, the features under contract, and for each feature the golden dataset (a fixed, versioned set of inputs with the verdict a domain expert gave each one) version and the model version each run, recorded under EV-RERUN, was pinned to. Version pinning is what lets one month be read against another: where we work in Braintrust, every change to a dataset is tracked so that an experiment can pin to a specific version. [Braintrust] Where a customer uses a different tool, we record the same two facts by hand.

2. State coverage before stating any score

The second section lists which features were covered this month and which were not, and why. A feature that was out of scope is named as out of scope rather than left off the page. This section is our own document rule, and it comes before the numbers on purpose: a reader who sees a score first tends to read it as a statement about the product rather than about the dataset.

3. List every run and what triggered it

One line per run: the date, the trigger, the dataset version, the model version, and the verdict. Triggers use the plan's own vocabulary: provider update, flagged prompt change, flagged retrieval (the step that fetches documents or records for the model to answer from) change, weekly schedule, and CI gate where the plan includes one. A run triggered by a provider announcement carries the change-log id and the provider's own source, so you can read what the provider said without leaving the report.

4. Show per-scorer deltas against the agreed thresholds

For each feature and each scorer, the report shows this month's result, the previous pinned result, and the difference. We read the difference, not the level, because a regression (a measured drop against the previous pinned run on the same dataset) is a change. Galtea publishes a tiered gate along the same lines: blocking failures such as safety or policy violations block a deploy at any rate, while regressions gate on the change in fail rate, with a faithfulness fail rate moving from 3 percent to 5 percent[src] as the worked example. [Galtea] That figure is Galtea's illustration, not a threshold we recommend, and it is sourced on the evidence page. Your thresholds are the ones agreed with you, and the report prints them beside the deltas so the comparison is visible rather than implied.

A threshold in this report is an agreement about what counts as a regression on your dataset. It is not a guarantee about your production traffic.

5. Report the judge's agreement with human verdicts, with more than one statistic

The judge is recalibrated against human verdicts at least monthly, which is one of the six written guarantees, and this section is where the month's result from EV-JUDGE is printed. Arize writes that a 75 to 90 percent[src] match between judge labels and human labels indicates strong alignment. [Arize AI] A raw match rate alone is the weaker statistic, so we do not print it alone. Hamel Husain writes that raw agreement can mislead when classes are imbalanced, and that the human labels should be treated as ground truth with the judge's true positive rate and true negative rate reported separately. [Hamel Husain] Galtea makes the same point with a worked pair: a judge at 0.9 accuracy and 0.1 Cohen's kappa is essentially guessing. [Galtea] The section also says why it exists every month rather than once: Future AGI writes that a judge model can move 3-5 points[src] on a fixed rubric (the written scoring standard a judge or a human labeller applies to an output) in a single month with no change on your side. [Future AGI]

6. Show what the production sample found against the dataset

Next, what the month's EV-PROD production sample showed: how many items were sampled, how they were scored, and where the production distribution differed from the dataset. Inputs that appeared in production and have no match in the dataset are listed by shape, and each one becomes a candidate case for the next dataset version. This section is a description of our own sampling work, and the sample size is the one written in your plan.

7. Say what the dataset does not cover

A separate section, in plain words, for the gaps: intents, languages, document types, tool paths and customer segments the dataset does not contain this month. This is the section most reports leave out, and it is the reason the numbers above can be read honestly. A gap listed here is not yet a finding. It is a statement that we cannot speak to that area, and it carries either a plan to close it or a note that closing it needs a decision from you.

8. End with recommended actions and the bounds of the report

Each finding gets one recommended action from a fixed set: rerun, re-pin (move the application to a specific model version instead of a moving alias), re-prompt or migrate. Each action names the finding it answers, the feature, and what it would take. Recommendations are recommendations: the shipping decision is yours, within the bounds on the what we cannot do page.

The report closes with its own bounds, in the same words every month. A score in this report is a statement about the dataset it was measured on, on the date it was measured, with the model version and dataset version printed beside it. It is not a measure of your product's quality in production, and it is not a claim about how the model will behave on inputs the dataset does not contain.

Outputs

• One monthly drift report per customer, delivered by the 5th business day.
• A coverage statement naming what was and was not covered, before any number.
• A run log with the trigger, the dataset version and the model version for every run.
• A per-scorer delta section printed beside the agreed thresholds.
• A judge agreement section carrying more than one statistic.
• A production sample section with the candidate cases it produced.
• A gaps section naming what the dataset does not cover.
• One recommended action per finding: rerun, re-pin, re-prompt or migrate.

Limits

Version and derivation

Version 1.0, derived from company definition, written 22 September 2026, by DriftCaught.