Skip to content
DriftCaught

What we cannot do

We cannot make a model deterministic or guarantee its outputs. We measure against an agreed dataset and thresholds and report what changed. We do not train or fine-tune models, and we do not replace human review for policy or legal decisions. Coverage is bounded by the dataset and the plan.

That paragraph appears on every plan page. This page explains each clause, because a limit you cannot check is not a limit.

  1. We cannot make a model deterministic

    A model can return a different answer to the same input, and a provider can change the weights behind a model id without changing your code. Temperature zero narrows the spread; it does not close it. Anyone who tells you otherwise is selling something we are not.

  2. We cannot guarantee outputs

    We guarantee a process: the dataset, the scorers, the calibration, the reruns, the report, and the six commitments on every plan. We do not guarantee what your model will say to the next user.

  3. We measure against an agreed dataset, not against reality

    A score is a statement about the dataset it was measured on. If your users start sending inputs the dataset does not contain, the score stays high while the feature gets worse. That is why production sampling exists, and it is also why the report tells you what the dataset does not cover.

  4. We do not train or fine-tune models

    We evaluate. Training and evaluation done by the same party on the same dataset is how you get a number that means nothing. If you want a model tuned, that is someone else's job, and we will happily evaluate the result.

  5. We do not replace human review for policy or legal decisions

    A scorer can tell you an output changed. It cannot decide whether an output is lawful, fair or acceptable to your regulator. Those decisions stay with people who are accountable for them.

  6. Coverage is bounded by the dataset and the plan

    A 500-item dataset does not cover everything a feature can be asked. A plan covering one feature says nothing about the other three you ship. Every report states the bound, and we would rather lose the sale than blur it.

  7. We do not publish measurements of any model

    The change log on this site restates what providers announced, with their sources. We run no public benchmark and we publish no score for any model. Our measurements belong to the customer whose dataset produced them.

  8. We cannot catch a regression the dataset has no case for

    If nobody ever wrote down that the summariser must not invent a date, no scorer will check it. The dataset grows from real failures because that is the only supply of cases that actually happen. Until a failure mode is in the set, it is invisible to the suite.

What we do commit to

Six things, on every plan, in writing. If a page on this site seems to promise a seventh, the page is wrong and we want to hear about it.

What we guarantee

  1. First eval baseline within 10 business days of access.
  2. Every tracked provider update triggers a rerun within 2 business days of the provider's announcement.
  3. Every flagged prompt or retrieval change is evaluated before it ships, within the plan's turnaround.
  4. Monthly drift report by the 5th business day.
  5. Judge recalibrated against human verdicts at least monthly.
  6. If a regression above the agreed threshold reaches production on a change we marked as passed, the next month is free.

These six are the whole list. Nothing else on this site is a guarantee.

Questions

So what am I paying for?
Ownership of a process that otherwise stops happening. Someone builds and grades the dataset, keeps the judge calibrated, reruns the suite when a provider ships, and writes down what moved. Those are the six guarantees, and they are the whole list.
If you cannot guarantee outputs, what does the regression guarantee mean?
It means this: if a regression above the threshold you agreed reaches production on a change we marked as passed, your next month is free. It is a statement about our work, not about the model.
Why publish this page at all?
Because the alternative is finding out in month three. A bounded service you can check is worth more than an unbounded promise you cannot.
Do you ever say a feature is fine?
We say the suite found no change above threshold on the dataset as it stands, and we say what the dataset does not cover. Those are different sentences and we do not collapse them.

Found a claim on this site we cannot support?

Write to us and we will correct the page or cut the claim. Every number here carries a source id you can check on the evidence page.

Related