EV-AGENTversion 1.0updated 2026-09-22

Span-level evaluation for agents

EV-AGENT scores one agent run at the level of its individual steps, so that a wrong tool call or a wrong decision path is visible even when the final answer reads well.

Applies to

Plans that run it

Derived from

Arize agent evals from traces, FutureAGI tool-selection metrics

Scope

This protocol covers any feature that calls tools, as an agent does, or fetches documents, as a RAG feature does, before it answers. It scores the run at the level of a span (one recorded step inside a request, for example a single tool call), rather than only at the level of the answer your user reads.

Scoring the answer alone inspects two endpoints, the input and the final output. Arize Phoenix writes that the interesting failures happen in the middle, in reasoning, tool selection and the decision path, where those two endpoints cannot show them. [Arize AI]

A score produced by this protocol is a statement about the dataset it was measured on. It is not a prediction about your production traffic, and we do not publish it.

Inputs

A written definition of one run: where it starts, where it ends, and which spans belong to it.
Traces (the ordered record of one run, every span with its timing, arguments and return) from your framework, with span names, tool arguments and tool returns.
A golden dataset (a fixed, versioned set of inputs with the verdict a domain expert gave each one) of agent cases, each carrying a human verdict.
The scorer (one automated check that returns a number or a label for one output) set agreed for this feature, and the active rubric (the written scoring standard an output is checked against, in your expert's words).
threshold (the number agreed with you at which a measured drop counts as a finding on this dataset) values, one set per resolution.

Steps

1. Fix what one run is, then capture its trace

Agree with you where a run begins and ends, and which spans belong inside it. Capture the trace from your framework rather than a log line per request. Future AGI writes that agent traces in current frameworks are routinely 15-40 spans[src] deep, so the record we ask for is a structure, not a pair of strings. [Future AGI]

Depth is published in spans, the unit the framework emits. We do not convert it into a step count, because no source publishes one.

2. Reconstruct the tool path and the tool inputs and outputs

From each trace we rebuild two signals. Arize Phoenix names them tool_path, the ordered tool calls with their arguments, and tool_io, what each tool was called with and what it returned. The same page warns that a correct-looking answer can come from a broken path, which is the case this protocol exists to find. [Arize AI]

3. Score the run at four resolutions

Future AGI describes scoring an agent at four resolutions: end-to-end, trajectory (the ordered path the agent took, judged as a path rather than by its last step), step and turn. The page warns in both directions. A high end-to-end score can hide bad tool calls, and strong per-step scores can still miss that the agent pursued the wrong goal. We record all four, and we report them separately rather than as one number. [Future AGI]

4. Choose a scorer for each kind of span

Different spans need different checks. LangSmith documents four evaluator types, from human review through code functions and LLM-as-judge (a model prompted with a rubric to score another model's output) to pairwise comparison of two application versions, and we pick per span type, following the selection method in EV-SCORE, rather than applying one check to the whole run. [LangChain] For a retrieval (the step that fetches documents or records for the model to answer from) span, Ragas defines faithfulness as the share of claims in a response that can be inferred from the retrieved context. That scores grounding in what was retrieved, not factual truth, so we use it on the retrieval span and not as a verdict on the run. [Ragas]

5. Add trajectory cases drawn from published failures

Agent failures that reach production are often path failures rather than wording failures. The AI Incident Database records incident 1152, dated 18 July 2025[src], in which an LLM-driven Replit agent reportedly executed destructive commands during a code freeze, deleted production data, generated fabricated records and incorrectly claimed rollback was impossible. That record is compiled from reported sources, not a vendor post-mortem, so we treat it as a shape of failure to write cases against, not as a confirmed engineering account. [AI Incident Database (Responsible AI Collaborative)] We add cases that put the agent near a destructive or irreversible action and check what the path does.

6. Record a human verdict beside the span scores

For every case in the dataset we store the domain expert's verdict next to the machine scores, at the span level where the expert marked a specific span. This is a filing step inside our service. Checking a judge against those verdicts is EV-JUDGE, not this protocol.

7. Write down the agreed thresholds, one set per resolution

We agree a number with you for each resolution and record it with the dataset version. A threshold here is an agreement about what counts as a regression (a measured drop against the previous pinned run on the same dataset) on this dataset. It is not a promise about your production traffic, and we do not present it as one.

8. Rerun on a trigger and pin the version

Your plan lists the triggers set out in EV-RERUN: a provider update, a flagged prompt change, a flagged retrieval change, the weekly schedule, and on the eight-feature plan your CI gate. On each trigger we run the dataset, pin the dataset version and the model version used, and write the verdict with one recommended action from rerun, re-pin (move the application to a specific model version instead of a moving alias), re-prompt or migrate. Span-level agent evals are in scope on the three-feature plan and above, and not on the single-feature plan.

Outputs

A scored run at four resolutions, end-to-end, trajectory, step and turn, reported separately.
A reconstructed tool path per case, with the failing span named and its arguments and return shown.
A per-scorer delta against the previous pinned run.
A written verdict with one recommended action: rerun, re-pin, re-prompt or migrate.
Any new case found in this run, labelled and written into the next dataset version.

Limits

What this protocol does not catch

A span score describes the cases in the dataset on the day they ran. It says nothing about traffic the dataset does not contain, and it does not predict production quality.
We can only score spans your framework emits. A step that is never recorded looks like a clean path, so missing instrumentation reads as a pass.
A clean tool path can still end in a wrong answer, and a good answer can come from a broken path. That is why all four resolutions are reported rather than one.
This protocol does not measure latency, cost, infrastructure failure or anything outside the recorded trace.
Cases written from published incidents cover failures other people reported. They do not cover the failure modes specific to your domain until a domain expert writes those cases.
Reruns reduce the chance that a change goes unnoticed. They cannot make a model behave the same way twice, and we do not claim they can.

Version and derivation

Version 1.0, derived from Arize agent evals from traces, FutureAGI tool-selection metrics, written 22 September 2026, by DriftCaught.

At a glance

Protocol: EV-AGENT
Version: 1.0
Updated: 2026-09-22
Citations: 7

Every citation resolves to an item on the evidence page, with the publisher, the URL and the date we read it.