Skip to content
DriftCaught

Start here

Eval baseline

$1,800 one-time, 10 business days. You get a golden dataset (a fixed, labelled set of inputs with the answers you consider correct), scorers, a calibrated judge, agreed thresholds and a first report. The fee is credited to your first month.

Every monthly plan reruns a suite. This is where the suite comes from. It is also the honest way to find out whether your feature has a measurable problem before you commit to a subscription.

Start the baselineSee the monthly plans

Ten business days, day by day

  1. Days 1 to 2

    Access and scope

    You send sample inputs and outputs, the prompt or pipeline, and a way to run it. We agree what counts as the feature and which failures matter to you. Nothing is built yet.

  2. Days 2 to 5

    Dataset from your failures

    We read your logs and your support queue for cases where the feature was wrong, and turn them into a labelled, versioned set. Your domain expert grades the ambiguous ones, because we do not know your domain better than you do.

  3. Days 5 to 7

    Scorers

    We pick the scorers that fit the feature type: deterministic checks where the answer has a right shape, similarity where it has a reference, a model judge where it needs reading. Each is chosen for what it catches, and we write down what it misses.

  4. Days 7 to 9

    Judge calibration

    The judge is scored against your expert's verdicts on the same items. We report the agreement figure. A judge that disagrees with your expert is a judge you should not gate a release on, and we say so rather than shipping it.

  5. Day 10

    Thresholds, report, walkthrough

    You agree the thresholds that gate a release. We hand over the first report with the current scores and the gaps, and walk you through it for 45 minutes. The suite runs in your stack and stays with you.

What you receive

  • Golden dataset built from your real failures and representative inputs, labelled and versioned
  • Scorers selected per feature type, judge calibrated against human verdicts with agreement reported
  • Thresholds agreed with you
  • First report with current scores and gaps
  • 45-minute walkthrough

Not included

  • Reruns after the baseline
  • Production sampling
  • Monthly reports

Scope and limits

Scope and limits
Features covered1
Turnaround10 business days
Ongoing rerunsNot included, a monthly plan covers them
Walkthrough45 minutes

Protocols run

What triggers a rerun

Nothing. The baseline is bought once and produces one report. It has no ongoing triggers, and saying otherwise would be the easiest thing on this page to get wrong.

A monthly plan is what adds them: a tracked provider update, a prompt change you flag, a retrieval change you flag, the weekly schedule, and on the platform plan a gate in your CI pipeline. Turnaround is 2 business days for a flagged change, or 1 on the platform plan, and a provider update reruns within 2 business days of the announcement on every plan.

EV-RERUN, the triggers and the procedure in full

After the ten days

Either you take a monthly plan and we own the reruns, or you keep the suite and run it yourself. It is built in your stack with your tooling, so it stays with you either way and nothing about it is locked to us.

What we guarantee

  1. First eval baseline within 10 business days of access.
  2. Every tracked provider update triggers a rerun within 2 business days of the provider's announcement.
  3. Every flagged prompt or retrieval change is evaluated before it ships, within the plan's turnaround.
  4. Monthly drift report by the 5th business day.
  5. Judge recalibrated against human verdicts at least monthly.
  6. If a regression above the agreed threshold reaches production on a change we marked as passed, the next month is free.

These six are the whole list. Nothing else on this site is a guarantee.

What we cannot do

We cannot make a model deterministic or guarantee its outputs. We measure against an agreed dataset and thresholds and report what changed. We do not train or fine-tune models, and we do not replace human review for policy or legal decisions. Coverage is bounded by the dataset and the plan.

The longer version, with the reasoning

Exit terms

Monthly. Cancel with 30 days' notice. No setup fee beyond the baseline. The baseline fee is credited to the first month of any plan started within 60 days.

Questions about the baseline

What does the baseline cost and how long does it take?
$1,800 one-time, 10 business days from access. The fee is credited to the first month of any plan started within 60 days.
What if we have no logged failures?
We build the first set from representative inputs you supply and from cases your support queue already knows about, and we mark which items came from production and which did not. The set grows from real traffic once the feature is running.
How many items are in the first dataset?
Enough to separate a real regression from noise on your feature, and small enough that a domain expert can actually grade it. Published guidance differs on where to start, and the honest answer is that your feature decides it, not a number we can quote. A set nobody has graded is not a golden dataset.
Can we do this ourselves?
Yes, and some teams should. The tools are good and they are documented. The reason to buy this is that dataset labelling, judge calibration and reruns are the work that quietly stops happening when a sprint gets busy. That is what we own.
What happens after day 10?
Either you take a monthly plan and we own the reruns, or you keep the suite and run it yourself. The suite is in your stack either way, and nothing about it is locked to us.
What counts as one feature?
A feature is one AI capability with its own prompt or pipeline and its own golden dataset. Two prompts behind one button are one feature. One prompt serving two products is one feature. A chat assistant and the summariser next to it are two.
Can I cancel?
Monthly. Cancel with 30 days' notice. No setup fee beyond the baseline. The baseline fee is credited to the first month of any plan started within 60 days.

A baseline is a measurement against a dataset you agreed to. It tells you where the feature stands today and what the suite can and cannot see. It does not tell you the feature is safe.

Related