Promptfoo vs DeepEval: choosing a starting point

Both are good, both are open source, and they are documented around different first moves. Promptfoo's documentation starts at the pipeline check that fails a build. DeepEval's starts at the dataset that is smaller than you think it needs to be. Pick the one whose first move matches your first problem.

What promptfoo documents

Promptfoo documents running evaluations in continuous integration to test prompt changes before they reach production and to fail the build when quality thresholds (the number a run has to stay inside before a change is allowed to ship) are not met. [Promptfoo]

That is a clear statement of purpose, and it fits a specific situation well: prompts live in a repository, changes arrive as pull requests, and somebody wants a bad edit stopped before it merges.

One note on reading that page carefully. It contains a pass rate figure inside an example snippet. It is an illustration of how the mechanism is configured, not a recommended threshold, and nobody should be quoting it as guidance. The number your build fails on comes from your own baseline and your own risk appetite, which is the argument in setting eval thresholds that gate a release.

What DeepEval documents

DeepEval's documentation says that even 10 to 20 inputs[src] is a great start for an evaluation dataset, and that production traces can be used to curate the next evaluation dataset. [Confident AI]

That is a useful thing for a vendor to publish, because the most common reason a team has no evals is the belief that starting requires hundreds of labelled rows. It also names a loop: run on a small set, look at production, curate the next set from what you find.

DeepEval also gives the category one of its more precise definitions. It defines a golden (a precursor to a test case that holds an input plus the expected result, and becomes a full test case once your application produces the actual output). [Confident AI]

Storing the input and the expected result separately from any output is what lets the same row be rerun against a new model version without being rewritten. Other tools use the word golden differently, which is why we attribute the definition when we use it. The wider question of starting sizes, where three published sources give three different answers, is in how many examples a golden dataset starts from.

The comparison that is honest

The centre of gravity differs. Promptfoo's documented emphasis is the gate. DeepEval's documented emphasis is the dataset and the metrics that run against it. Both projects have moved well beyond those starting points, so for any capability not cited on this page, read the current docs rather than a comparison post.

What we will not do is publish a feature matrix from memory. A stale row in a comparison table is a misrepresentation of somebody's work, and these projects ship faster than any article about them.

Both tools are open source and both can be trialled in an afternoon on one feature. That makes most of the written comparison unnecessary, including this one, once you have an hour to spend.

The part that is the same in both

Underneath, the structure does not vary by tool.

Weights and Biases documents an evaluation as a dataset of test examples plus one or more scoring functions, used to measure performance against repeatable test cases, compare changes over time and identify regressions (a case that passed on the previous run and fails on this one). [Weights & Biases]

LangSmith documents four evaluator types[src]: human review, rule based code functions, LLM as judge (a model that scores another model's output against those rules) in reference free and reference based variants, and pairwise comparison of two versions of an application. [LangChain]

And for features that answer from fetched material, Ragas defines faithfulness as the share of claims in a response that can be inferred from the retrieved context, expressed as supported claims divided by total claims[src]. [Ragas] Read that definition before adopting it: it scores grounding in the material supplied by retrieval (the step that fetches documents or passages for the model to answer from), not whether that material was true.

Those three ideas travel between tools. Our own scorer selection method is EV-SCORE, and it names the tool only where the tool's own documentation is the source.

A way to decide today

Two questions settle it for most teams.

What broke last? If the answer is a prompt edit that shipped and made things worse, start where the CI documentation is, because you need a check in the path the change takes. That situation is written up in a prompt edit regressed the output.

What do you not know? If the answer is how the feature behaves at all, start where the dataset documentation is, because a gate with nothing behind it blocks on noise.

Either way the decision is cheap to reverse. The expensive artifacts are the labelled dataset, the written rubric (the written rules applied to decide whether an output passes) and the calibration record for your judge. Those are yours, they are portable, and they are what actually determines whether any of this catches anything.

What no tool in this category does for you

Neither tool watches four providers' deprecation and changelog pages on your behalf and works out which of your pinned models is about to move. That is a reading job on a schedule, and it is what the model change log exists to be.

Neither tool decides which person's verdict settles a disagreement about a borderline output, and neither keeps rechecking that your judge still agrees with that person a month later.

Where the line sits between buying software and buying that ownership is the subject of eval tool or eval service. If you want the dataset and the calibration built once against your own traffic, that is the eval baseline.