Eval tool or eval service: which one you need

If one person on your team owns the dataset, the thresholds (the numbers a run has to stay inside before a change is allowed to ship) and the judge (a model that scores another model's output against written rules) calibration, and has a few days a month for them, buy a tool and stop reading. What we sell is ownership of that work, not better software, and for a lot of teams the tool is the correct answer.

What a tool gives you

The tools in this category are good and their documentation is clear about what they do.

Braintrust recommends running evals on every pull request to catch regressions before they reach production. [Braintrust] Langfuse says observation level evaluators are the recommended target for live production data and lists sampling a percentage of traces as a way to manage the cost of judge based evaluation. [Langfuse]

Those are real capabilities, they work, and we use tools like them every day. A comparison of two of them on their own documented terms is in Braintrust vs Langfuse.

Software gives you somewhere to put a golden dataset (a fixed set of saved inputs with the expected result for each one, which an eval run scores against), a way to run scorers against it, a place for the results, and a hook into your pipeline. That is most of the machinery.

What a tool does not give you

The machinery is not the expensive part. Four pieces of work sit outside it, and none of them is a feature anyone can ship.

Someone has to decide what good looks like

Hamel Husain writes that in most organisations there are one or two key individuals whose judgment is crucial to the AI product, and that a judge is aligned iteratively against that person's own critiques. [Hamel Husain]

That person works for you and no vendor can replace them. What can be organised is the part around them: running the labelling sessions, turning critiques into written rules, chasing the ambiguous cases back to them, and keeping the record.

The calibration set has to be big enough to conclude anything

Hamel Husain recommends about 100 examples per failure mode[src] to validate an automated judge, and writes that below 60 examples the confidence intervals are often too wide to support a useful conclusion. [Hamel Husain]

Per mode. With four failure modes, that is a labelling programme rather than an afternoon, and it is the step most often skipped, which is how teams end up with an agreement figure that means nothing. The sizing rules are in EV-DATA.

The instrument moves while you are not looking

Future AGI writes that a judge model can move 3 to 5 points[src] on a fixed rubric in a single month without any change on your side. [Future AGI] That range is Future AGI's published figure, not something we have measured. Future AGI also writes that OpenAI, Anthropic, Google and most frontier providers update model weights without a major version bump, which is that vendor's characterisation of the market rather than a provider statement. [Future AGI]

Nothing alerts you to this. Recalibration has to be scheduled and done, which is why EV-JUDGE puts it on a fixed cycle rather than on a trigger.

The dataset ages

Future AGI recommends refreshing a golden dataset monthly by default, sampling 200 to 500[src] recent production traces stratified by intent and persona, labelling them against the active rubric and versioning the result as a new artifact. [Future AGI]

And Galtea warns against updating ground truth to match model behaviour, recommends code review on additions and treats deletions as a production risk. [Galtea] That is a governance habit, held every month by someone who cares about it, not a setting.

Buy the tool when this is true

Be honest against this list. If most of it holds, a subscription to us is a waste of your money.

• A named person owns evals and it is in their objectives, not in their spare time.
• Labelling sessions with your domain expert already happen on a schedule.
• Your judge has a dated calibration record and somebody could produce it this afternoon.
• Someone reads the provider deprecation and changelog pages weekly and knows which of your features are affected.
• The dataset got a new version in the last two months for a reason somebody wrote down.

Teams where that is true do not need us. They need budget for the tool and protection for the person's time.

Buy the service when this is true

The other pattern is just as common, and it usually looks like this: the tool is installed, there are forty rows in a dataset from a sprint eight months ago, the judge was never checked against human labels, and nobody can say what changed when the provider shipped a new version last week.

That is not a tooling failure. It is the ordinary result of asking a team with a product roadmap to also hold a measurement discipline every month.

What we take on is the ownership: the dataset and its refresh, the scorer set, the judge calibration with its statistics and label counts, the thresholds agreed with you and written down, the reruns when a provider announces something, and the monthly report. The triggers and turnarounds are in EV-RERUN, and you can read a sample report before talking to anyone.

The first piece of work is a one time eval baseline at 1,800 dollars, which builds the dataset, selects the scorers, calibrates the judge and agrees the thresholds. The commitments we make and the limits on each plan are listed on the plans page.

The claim we are not making

We are not saying tools miss things that we catch, and we publish no measurement of any model to support such a claim. The tools do what their documentation says. The difference is who holds the dataset, the thresholds and the calibration between one release and the next, and whether that person has the month to do it.

If the answer is somebody on your side, keep it there. If the answer is nobody, that is the gap this exists to fill, and the contact page is the shortest route to finding out which case you are in.