A retrieval (the step that fetches documents or passages for the model to answer from) augmented feature answers from documents you fetch for it. That gives it two places to fail, and the two fail independently. A single answer quality number tells you something got worse without telling you which half to look at.
What typically breaks in RAG after a model change
Five patterns, and the first is usually not the provider's doing at all. Each shows up as drift (a measured change in a feature's scores between two runs of the same dataset).
The right documents stop coming back
retrieval failures are the ones people expect, and they are usually not caused by a model update at all. A reindex, an embedding model change, a chunking change, a filter added for a new tenant. The generation side is blameless and the answer is still wrong, because the passage that held the answer never reached the prompt.
This is why a golden dataset (a fixed, versioned set of inputs with the verdict a domain expert gave each one) for RAG stores the expected source document per question, not only the expected answer.
The same context gets used differently
This is the failure specific to a model change, and the one that surprises teams. Retrieval returns exactly what it returned last month. The answer changes anyway. Which passage the model leans on, how it reconciles two passages that disagree, whether it answers from a partial match or declines: those are model behaviours, and a new version weighs them differently.
Nothing on your side has to move for this to happen. Google says a Gemini latest alias points to the most recent release of a model variation and is hot-swapped with every new release of that variation. [Google] Google also says stable models usually don't change and that most production apps should use a specific stable model, which is the provider's own argument for pinning. [Google] Keep the hedge in that sentence: usually is not never. Dated entries are in the Google change log.
Answers drift away from the retrieved context
Claims appear in the answer that the retrieved passages do not support. Ragas defines faithfulness as the share of claims in a response that can be inferred from the retrieved context, computed by identifying every claim and checking each one against the context, expressed as supported claims divided by total claims[src] on a 0 to 1 scale. [Ragas]
Read that definition before you agree to it as your headline metric. Faithfulness scores grounding in the context that retrieval returned. It does not score whether that context is true.
Citations stop matching the sentences they sit next to
If your interface shows sources, the link between a claim and its citation is a separate thing to check. An answer can be faithful in aggregate and attach the wrong passage to a specific sentence. Users trust the citation more than the sentence, so this failure costs more than its score suggests.
Scoring only the final answer hides all of the above
Arize Phoenix writes that end-to-end evaluation only inspects the two endpoints, the input and the final output, and that the interesting failures happen in the middle, in reasoning, tool selection and the decision path, where those two endpoints cannot show them. [Arize AI] For RAG the middle is the retrieval step and what the model did with what it got back. The method for reading it is EV-AGENT, which applies to retrieval augmented features as well as to agents.
Which scorers apply
LangSmith documents four evaluator types: human review, rule based code functions, LLM as judge (a model that scores another model's output against a written standard) in reference free and reference based variants, and pairwise comparison of two application versions. [LangChain] RAG uses code checks on the retrieval side, where there is an expected document, and judged scorers (the functions that turn one output into a score or a pass and fail verdict) on the generation side, where there is not.
- Faithfulness against the retrieved context. Catches claims the passages do not support. Misses whether the passages are correct, and misses a right answer that was reached without the context.
- Retrieval hit rate against the expected document. Catches the passage that should have been fetched and was not. Misses cases where the right document was fetched and the wrong part of it was used.
- Answer relevance judge scorer. Catches an answer that is grounded but does not address the question asked. Misses factual error, and needs calibrating against your people's labels before it is reported.
- Citation validity code check. Catches a citation pointing at a document that was not retrieved, or at a chunk that no longer exists. Misses a valid citation attached to the wrong sentence unless the dataset records the expected span (one recorded step inside a request, for example a single tool call or retrieval, with its inputs and outputs).
- Reference based answer match. Catches drift away from a known correct answer, on the subset of questions that have one. Misses everything on open questions, which is most of a real corpus.
The split matters more than any single number. A verdict that says retrieval held and generation moved sends you to the prompt. A verdict that says retrieval moved sends you to the index. One combined score sends you nowhere. Selection is written out in EV-SCORE.
What a baseline includes for this feature type
The eval baseline for a RAG feature is built around question and source pairs.
- Real questions from your own traffic, including the ones your corpus cannot answer, because declining well is a behaviour worth scoring.
- The expected source document or chunk for every question where one exists, stored alongside the question, so retrieval can be scored on its own.
- A claim-level rule for what counts as supported by the context, agreed in writing before any faithfulness number is produced.
- A snapshot of the corpus version and the retrieval configuration pinned to the dataset version, so a later run can tell a retrieval change apart from a model change.
- Questions your corpus genuinely cannot answer, with the expected behaviour written down.
- A dataset change rule. Galtea writes that additions should require code review, that deletions should be treated as production risk, and warns against updating ground truth to match model behaviour. [Galtea] Construction is EV-DATA.
Which plan fits
Up to three features at $4,400 a month is the plan we point RAG features at, because it includes span-level evals and RAG is the feature type where the endpoints lie to you. You can run a RAG feature on One feature, and you will get faithfulness, retrieval hit rate and the rerun triggers. What you will not get there is per-span scoring of the retrieval step, which starts on the three-feature plan. The other caveat is scope: production sampling scores live traffic against reference free scorers only, since no expected answer exists there. Langfuse says observation-level evaluators are the recommended target for live production data and lists sampling a percentage of traces as a way to manage the cost of judged evaluation. [Langfuse] That mechanism is EV-PROD. The same grounding question, asked without retrieval, is on the summarisation hub.