A chat assistant is the feature type where the gap between a demo and production is widest. One prompt in a playground looks fine. Twelve turns with a frustrated user, a pasted invoice and a question about a refund is a different test, and it is the one that fails.
What typically breaks in a chat assistant after a model change
Five patterns come up often enough to belong in every dataset we build for this feature type.
A rule holds for the first turns and then stops holding
Your system prompt carries rules. Never quote a price. Ask for an order number first. Stay in the product's voice. A new model version weighs those rules against the user's most recent message differently, and the user's framing accumulates with every turn. Single-turn tests keep passing because the rule is still fresh at turn one.
Nothing on your side has to change for this to start. Google says a Gemini latest alias points to the most recent release of a model variation and is hot-swapped with every new release of that variation. [Google] Future AGI writes that OpenAI, Anthropic, Google and most frontier providers update model weights without a major version bump. [Future AGI] That is Future AGI's characterisation of the market, not a provider statement.
The assistant answers a question it should have handed off
Under-refusal is the expensive direction. The assistant gives a confident answer on a policy, legal or eligibility question that your written rules say a human handles. The Markup reported on 29 March 2024 that New York City's MyCity chatbot told users landlords do not need to accept Section 8 vouchers and that an employer can take a cut of workers' tips, both of which are unlawful. [The Markup] That is a journalistic investigation rather than an operator's incident report, so treat it as what was reported, not as a diagnosed root cause.
The reverse direction costs you too. An assistant that starts refusing ordinary questions after an update looks broken to your users and produces no error in your logs.
Tone and length drift without any model swap
A system prompt edit of your own can move the feature as far as a provider update does. In a postmortem published on 23 April 2026, Anthropic attributed six weeks of Claude Code quality reports to three product-layer changes and measured a 3 percent quality drop[src] from one system prompt change on its own eval suite. [Anthropic] Anthropic states that the API was not impacted and model weights were unchanged. The point for you is narrower and more useful: a prompt change is a change, and it needs a run.
Context handling across a long conversation shifts
Which parts of a long conversation the model leans on is not something you control directly. After an update the assistant may re-ask for the order number it was already given, or answer from a detail the user corrected six turns earlier. This is a drift (a measured change in a feature's scores between two runs of the same dataset) pattern that only a multi-turn dataset row can surface, because the failing behaviour needs earlier turns to exist at all.
The judge that scores the assistant moves too
Most of what matters in a chat assistant is scored by a judge (a model that scores another model's output against a written standard), and the judge is itself a hosted model. Future AGI writes that a judge model can move 3-5 points[src] on a fixed rubric (the written scoring standard applied to an output, in your expert's words) in a single month with no change on your side. [Future AGI] A score that drops without a recalibrated judge behind it is not yet a finding about your assistant.
Which scorers apply
LangSmith documents four evaluator types: human review, rule based code functions, LLM as judge in reference free and reference based variants, and pairwise comparison of two application versions. [LangChain] A chat assistant uses all four, because its failure modes split cleanly into checkable ones and arguable ones. Each scorer (one automated check that turns an output into a number or a pass and fail verdict) below is written with its own limit, and the limits go in the report.
- Rule following judge scorer. Catches a broken rule at the turn it breaks, when the rule is written into the rubric. Misses any rule nobody wrote down, and cannot tell you whether the rule was reasonable.
- Refusal behaviour code check. Catches hand-offs that did not happen, when the case has a written expected outcome. Misses a refusal that is correct but rude, and misses novel questions your policy list does not name.
- Tone and persona rubric judge scorer. Catches a voice sliding away from your product. Misses factual errors entirely, and its verdicts are only worth reporting once it has been calibrated against your people's labels.
- Multi-turn context retention check. Catches a detail from turn two that vanished by turn eight. Misses cases where the model kept the detail and used it wrongly.
- Pairwise comparison. Catches which of two prompt versions your expert prefers when no absolute scale exists yet. Gives you no standing number, so it never replaces the scored set.
Raw agreement is the wrong way to report any of these. Hamel Husain writes that agreement can mislead when classes are imbalanced, and that human labels should be treated as ground truth with the judge's true positive rate and true negative rate reported separately. [Hamel Husain] Galtea makes the same point with numbers: a judge showing 0.9 accuracy with 0.1 Cohen's kappa[src] is essentially guessing. [Galtea] The full procedure is in EV-JUDGE, judge calibration.
What a baseline includes for this feature type
The eval baseline for a chat assistant produces a golden dataset (a fixed, versioned set of inputs with the verdict a domain expert gave each one) shaped for conversations, not for prompts.
- Whole conversations as rows, sampled from your real traffic, each carrying the turn at which the assistant is judged and the turns that set it up.
- Every rule in your system prompt turned into an explicit label rule, so a broken rule is a named failure mode rather than a low score.
- A refusal set with both directions in it: cases that must be handed off, and ordinary cases that must not be.
- A persona and tone rubric written in your expert's own words. Hamel Husain writes that most organisations have one or two key individuals whose judgment is crucial to the product, and that the judge is aligned iteratively against that person's critiques. [Hamel Husain] That person is yours, not ours.
- A judge calibrated against those human labels, with more than one agreement statistic reported, and a recalibration cycle set.
- threshold (the number agreed with you at which a measured drop counts as a finding on this dataset) values agreed with you per scorer, including which drops block a release. See EV-THRESH, thresholds and gates.
Dataset construction itself is written out in EV-DATA, golden dataset construction.
Which plan fits
One feature at $1,900 a month fits a single chat assistant in production. It covers reruns on every tracked provider update, on every prompt change or index change you flag, and weekly, plus production sampling and the monthly report. Two honest caveats. If your assistant calls tools, scoring the reply alone will not see a bad tool call, and evals at the level of a span (one recorded step inside a request, for example a single tool call, with its inputs and outputs) start on the three-feature plan, which the agent hub explains. And a threshold is a number agreed with you on your dataset, not a promise about your production traffic. The bounds are listed in what we cannot do, and the updates we track are in the model change log.