Production sampling and drift detection

Scope

A scheduled rerun scores a fixed dataset. Real traffic does not stay fixed. This protocol covers the other half: sampling what your feature actually handled in production, scoring that sample against the rubric (the written grading instructions an output is checked against) agreed at baseline, and reading the result against the previous period so drift (a change in how a feature behaves over time, measured against an agreed dataset and rubric) shows up between scheduled runs.

It covers sampling, scoring and reporting. It does not set thresholds (the number agreed with you at which a measured drop counts as a finding on this dataset), which are agreed with you under EV-THRESH, and it does not replace the reruns in EV-RERUN.

Production scores are scores against your rubric, on your traffic. They are not a measurement of any model, and we publish none.

Inputs

• Read access to traces or logs for the tracked feature, for the period being sampled.
• The scorers and the rubric agreed at baseline, at a recorded version.
• The current golden dataset (a fixed set of saved inputs with the expected result for each one, which an eval run scores against) version and the scores from its most recent run.
• The thresholds agreed with you, and the escalation path for crossing one.
• Your plan's monthly production sample allowance.

Steps

1. Agree what is sampled, and at what rate

We agree with you which feature, which route and which period are in scope, then set a sampling rate against the allowance on your plan: 1,000 traces a month on One feature, 5,000 on Up to three features, 20,000 on Up to eight features. Langfuse says observation-level evaluators are the recommended target for live production data, and lists sampling a percentage of traces as a way to manage the cost of LLM-as-a-judge (a model prompted with your rubric to grade another model's output) evaluation.[Langfuse] That page publishes no recommended percentage, so we do not quote one, and it is sourced on the evidence page. Your rate comes from your traffic, your allowance and what you want covered.

2. Capture the sample

We pull traces across the whole period rather than in one burst, so a single busy hour does not stand in for the month. Each captured trace keeps its input, its final output and the span (one recorded step inside a single request, for example a document fetch or a tool call) records in between, with the model id, the prompt version and the timestamp. We store the sample at a version, the same way a dataset is stored, so a later reading of the same period looks at the same traces.

3. Score the sample with the agreed scorers

The sample is scored with the scorers already agreed for that feature type, not a fresh set chosen after seeing the traffic. LangSmith documents four evaluator types: human review, rule-based code functions, LLM-as-judge in reference-free and reference-based variants, and pairwise comparison of two application versions.[LangChain] Most production traces have no reference answer, so the working mix is usually code checks plus a reference-free judge, with human review on the calibration sample.

4. Score inside the trace, not only at the two endpoints

Arize Phoenix writes that end-to-end evaluation only inspects the two endpoints, the input and the final output, and that the interesting failures happen in the middle where they cannot be seen, naming reasoning, tool selection and the decision path.[Arize AI] So where a feature retrieves documents, as a RAG feature does, or calls tools, as an agent does, we score the middle as well: whether retrieval (the step that fetches documents or records and puts them into the prompt) returned the right material, whether the right tool was chosen, and whether the final answer used what came back.

5. Read the change, not the absolute number

A single month's fail rate on its own is hard to act on. Galtea recommends a tiered release gate in which blocking failures such as safety or policy violations trigger a deploy block at any rate, while regressions (a case that passed in the previous period and fails in this one) gate on the change in fail rate, giving the example of a faithfulness fail rate rising from 3 percent to 5 percent[src].[Galtea] Those figures are Galtea's illustrative example, not a recommended threshold. We use the shape of it: policy and safety failures are raised at any rate, and everything else is read as a regression against the previous period, against the threshold you agreed.

6. Feed what production found back into the dataset

A sample earns its keep when the new failure modes it surfaces become dataset items. Future AGI recommends refreshing a golden dataset monthly[src] by default, sampling 200 to 500[src] recent production traces stratified by intent and persona, labelling them against the active rubric, and versioning the dataset as a new artifact; the same page gives weekly refreshes during rapid change and a refresh immediately after model or corpus updates.[Future AGI] We propose candidates against that shape and you approve them before anything enters the dataset built under EV-DATA.

7. Hold the line on ground truth

Galtea writes that a golden dataset drifts as product scope changes and failure modes are discovered, that additions should require code review, that deletions should be treated as production risk, and it warns against updating ground truth to match model behaviour.[Galtea] We follow that warning literally. When a scored trace disagrees with the expected output, we either confirm the expected output or change it because the product changed, and we record which of the two happened. We never edit the expected output to agree with what the model did.

8. Record the period and report it

We write the period up once: the sample size and rate, the scores by scorer, the change against the previous period, every new failure mode with example traces, and the dataset candidates proposed. You receive the monthly drift report by the 5th business day, which is one of the six written guarantees. Anything that crosses an agreed threshold does not wait for the report: it is raised within your plan's turnaround, and it can start a rerun under EV-RERUN.

Outputs

• A scored, versioned sample of production traces for the period.
• Scores by scorer, with the change against the previous period.
• New failure modes, each with example traces and the spans involved.
• Candidate dataset items, proposed for your approval.
• The monthly drift report, and an escalation when a threshold is crossed.

Limits

Version and derivation

Version 1.0, derived from Galtea online evaluation, Arize Phoenix, Langfuse, written 22 September 2026, by DriftCaught.