Every externally sourced number and claim on this site, with its publisher, its URL and the date we read it. 45 items, verified 22 September 2026. If a figure elsewhere on the site has no id here, it should not be on the site.
We publish this because we ask you to trust a service you cannot watch. The least we can do is let you check the parts you can.
Change-log entries restate provider announcements with sources and dates. Impact notes cite public practitioner reports. The site publishes no measurements of its own on any model.
15 items
Release notes, deprecation tables and model pages published by the provider itself. These are what the change log rests on.
Anthropic says it gives customers with active deployments at least 60 days' notice before retiring a publicly released model.
Phrase as Anthropic's own published commitment for the Claude API and Anthropic-operated platforms. The page states partner platforms (Amazon Bedrock, Google Cloud) set their own schedules, so do not present 60 days as an industry norm or extend it to other providers.
verified 2026-09-24
Anthropic describes its model lifecycle in four states, active, legacy, deprecated and retired, and says requests to retired models will fail.
Use for the change log vocabulary. Attribute the four state names to Anthropic; other providers use different words, so do not present these as universal terms.
verified 2026-09-24
Anthropic notified developers on 14 April 2026 that Claude Sonnet 4 and Claude Opus 4 would be retired, and both were retired on 15 June 2026.
A worked example of a dated retirement schedule, roughly two months between notice and retirement. State both dates. Do not describe the gap as a guarantee; the published commitment is the separate 60-day notice item.
published 2026-04-14, verified 2026-09-24
Anthropic notified developers on 5 June 2026 that Claude Opus 4.1 would be retired on 5 August 2026, with Claude Opus 4.8 as the recommended replacement.
Use as a change log entry and as an example of a forced migration with a named replacement model. Do not claim anything about how the replacement behaves differently; no measurement is published here.
published 2026-06-05, verified 2026-09-24
Anthropic deprecated the temperature, top_p and top_k request parameters from Claude Opus 4.7 onward, and a non-default value now returns a 400 error.
Use as an example of a provider change that breaks caller code without any model swap by the customer. Report it as a parameter deprecation only; do not infer any effect on output quality.
verified 2026-09-24
Anthropic's own release notes record Claude Opus 4.8 on 28 May 2026, Claude Opus 5 on 24 July 2026 and Claude Fable 5.1 on 1 September 2026.
Use to support the cadence claim with the provider's own dated entries. Cite the specific launches and dates; do not state an average interval or project a future cadence.
Anthropic: Claude Platform API release notes
published 2026-09-01, verified 2026-09-24
OpenAI says it gives at least 6 months' notice before retiring a generally available model and at least 3 months for specialized variants. For preview models it commits to no minimum: the page says they may be retired with much shorter notice, such as 2 weeks, and that safety or compliance concerns may require faster timelines than any of the stated periods.
Keep the tiers together. The 6 months and 3 months are stated minimums; the 2 weeks is OpenAI's own illustration for preview models and must never be written as a floor or as "at least". Quoting only the 6 months overstates the commitment for preview models. Attribute to OpenAI's own deprecations page.
verified 2026-09-24
OpenAI announced on 11 June 2026 that the gpt-5-2025-08-07 snapshot would be shut down on 11 December 2026.
Use as a change log entry and as evidence that pinning to a dated snapshot buys time but not permanence. State both dates exactly.
published 2026-06-11, verified 2026-09-22
OpenAI's API changelog records GPT-6 Astra on 3 September 2026, GPT-Live 1 on 10 September 2026 and GPT Image 2.5 models on 8 September 2026.
Cite the dated entries as published. Do not extrapolate a release rate or claim any of these changed behaviour for a customer's feature.
published 2026-09-10, verified 2026-09-22
Google says a Gemini "latest" alias points to the most recent release of a model variation and is hot-swapped with every new release of that variation.
The strongest provider-own evidence that behaviour can change with no customer action. Phrase it as Google's documented alias behaviour, not as a claim that any specific feature regressed.
verified 2026-09-22
Google says Gemini preview models may be used in production but will be deprecated with at least 2 weeks' notice.
Use when explaining why a feature pinned to a preview model needs a shorter rerun cycle. Quote the 2-week figure exactly and keep it scoped to Gemini preview models.
verified 2026-09-22
Google's Gemini documentation says stable models usually don't change and that most production apps should use a specific stable model.
Use to support the re-pin recommended action. Keep Google's hedge, "usually don't change"; do not upgrade it to a guarantee that stable models never change.
verified 2026-09-22
Google's Gemini API changelog records gemini-3.5-flash reaching general availability on 19 May 2026, gemini-3.7-flash on 13 August 2026 and gemini-3.8-flash on 2 September 2026.
Cite the dated entries as published. Do not present the interval between them as a predicted cadence.
published 2026-09-02, verified 2026-09-22
Google announced on 2 April 2026 that the retirement dates for Gemini 2.5 Pro, Gemini 2.5 Flash and Gemini 2.5 Flash-Lite on Vertex AI had been updated to 16 October 2026.
Use as evidence that published retirement dates themselves move. Say the dates were updated; do not characterise the change as shortened or extended without the earlier date in hand.
Google: Generative AI on Vertex AI release notes
published 2026-04-02, verified 2026-09-22
Meta announced Llama 4 Scout and Llama 4 Maverick on 5 April 2025 as open-weight natively multimodal models available to download the same day.
Meta's own announcement. Use for the Llama section of the change log. Note that open-weight models are pinned by the customer, so the rerun trigger is the customer's upgrade, not a provider hot-swap.
Meta: The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation
published 2025-04-05, verified 2026-09-22
23 items
Published guides by people and companies who do this work. These are what the protocols cite, step by step.
Future AGI describes scoring an agent at four resolutions, end-to-end, trajectory, step and turn, and writes that a high end-to-end score can hide bad tool calls.
Primary support for EV-AGENT. The page also warns that strong per-step scores can still miss that the agent pursued the wrong goal, so present both directions of the failure, not only the end-to-end one.
Future AGI: Multi-Mode Agent Evaluation
verified 2026-09-22
Future AGI recommends refreshing a golden dataset monthly by default, sampling 200-500 recent production traces stratified by intent and persona, labelling them against the active rubric, and versioning the dataset as a new artifact.
Cite when EV-DATA or EV-PROD describes refresh cadence and sample size. Keep the 200-500 range intact. The page also gives weekly during rapid change and immediately after model or corpus updates; include that qualifier if the monthly figure is quoted.
Future AGI: LLM Eval Data Drift Detection: Three Drifts That Age Your Golden Set
published 2026-03-03, verified 2026-09-22
Hamel Husain writes that his heuristic is to start with around 30 examples and keep going until he sees no new failure modes.
This is the starting figure the site may cite for dataset construction. Quote it as "around 30" and attribute it to Hamel Husain. Do not restate it as 50 or as a DriftCaught standard.
Hamel Husain: Creating a LLM-as-a-Judge That Drives Business Results
verified 2026-09-22
Hamel Husain recommends about 100 examples per failure mode to validate an automated judge, and writes that below 60 examples the confidence intervals are often too wide to support a useful conclusion.
Use in EV-JUDGE for how large a calibration set must be before an agreement number means anything. Keep both figures together; the 60 floor is the point of the claim.
Hamel Husain: Creating a LLM-as-a-Judge That Drives Business Results
verified 2026-09-22
Hamel Husain writes that raw agreement can be misleading when classes are imbalanced, and that the human labels should be treated as ground truth with the judge's true positive rate and true negative rate reported separately.
Cite whenever the site reports a judge agreement statistic. The worked example on the page is that a judge which always predicts Pass still scores 95% agreement when errors occur in 5% of examples.
Hamel Husain: Creating a LLM-as-a-Judge That Drives Business Results
verified 2026-09-22
Hamel Husain writes that in most organisations there are usually one or two key individuals whose judgment is crucial to the AI product, and that the judge is aligned iteratively against that person's own critiques.
Primary support for the claim that a named domain expert grades the seed set. Phrase it as the practice Hamel Husain describes, and make clear the expert is the customer's, not ours.
Hamel Husain: Creating a LLM-as-a-Judge That Drives Business Results
verified 2026-09-22
Arize's LLM-as-a-judge guide tells practitioners to start by pulling 50 real traces from their system.
The nearest published figure to the seed claim. Phrase it exactly as Arize does, "pull 50 real traces", not "50 real failures". The page does not say the 50 must be failures or that a domain expert grades them, so do not add either.
Arize AI: LLM as a Judge: Primer and Pre-Built Evaluators
verified 2026-09-22
Arize writes that a 75 to 90 percent match between judge labels and human labels indicates strong alignment and readiness to scale, and that its pre-built evaluators achieve target precision and F-scores above 70 percent.
Use in EV-JUDGE as one published alignment bar, attributed to Arize. Pair it with the Hamel item on true positive and true negative rates, because raw match rate alone is the weaker statistic.
Arize AI: LLM as a Judge: Primer and Pre-Built Evaluators
verified 2026-09-22
Arize Phoenix writes that end-to-end evaluation only inspects the two endpoints, the input and the final output, and that the interesting failures happen in the middle where they cannot be seen.
This is the citation for the claim that end-to-end scoring misses tool and retrieval regressions. The page names reasoning, tool selection and decision path as the hidden failures; list those rather than inventing others.
Arize AI: Trace-level Evaluation: Beyond Input/Output Checks
verified 2026-09-22
Arize Phoenix describes reconstructing two signals from an agent trace, tool_path, the ordered tool calls with their arguments, and tool_io, what each tool was called with and what it returned, and warns that a correct-looking answer can come from a broken path.
Primary method citation for EV-AGENT. Use the published names tool_path and tool_io when describing what a span-level eval reads.
Arize AI: Trace-level Evaluation: Beyond Input/Output Checks
verified 2026-09-22
Braintrust says every change to a dataset is tracked so that experiments can pin to specific versions.
Cite for dataset versioning in EV-DATA. Braintrust's docs describe their product's behaviour, so phrase it as what the tool provides, not as a general industry practice.
verified 2026-09-22
Braintrust recommends running evals on every pull request to catch regressions before they reach production.
Cite in EV-THRESH and on the platform plan's CI gate section. Attribute the recommendation to Braintrust.
verified 2026-09-22
Galtea recommends a tiered release gate in which blocking failures such as safety or policy violations trigger a deploy block at any rate, while regressions gate on the change in fail rate, giving the example of a faithfulness fail rate rising from 3 percent to 5 percent.
Use in EV-THRESH as one published gating design. The 3 to 5 percent figure is Galtea's illustrative example, not a recommended threshold, so present it as an example.
Galtea: Automated LLM Evaluation: Building a CI/CD quality gate that actually runs
published 2026-07-01, verified 2026-09-22
Galtea writes that a golden dataset drifts as product scope changes and failure modes are discovered, that additions should require code review and deletions be treated as production risk, and warns against updating ground truth to match model behaviour.
Strong citation for the dataset governance section of EV-DATA and for the drift report. Quote the warning about updating ground truth to match model behaviour; it is the clearest published statement of the rule.
Galtea: Automated LLM Evaluation: Building a CI/CD quality gate that actually runs
published 2026-07-01, verified 2026-09-22
Galtea writes that a judge must be run against a labelled gold set, with the gap between its verdicts and the human labels measured and the prompt iterated until the gap closes.
Cite as the calibration procedure in EV-JUDGE. The same page recommends versioning judge prompts alongside application code, which the site may cite in the same paragraph.
Galtea: LLM as a Judge: The Complete Guide
published 2026-05-07, verified 2026-09-22
Galtea recommends measuring a judge with accuracy, precision, recall, F1, Pearson correlation, Spearman correlation and Cohen's kappa rather than one statistic, and writes that a judge with 0.9 accuracy and 0.1 Cohen's kappa is essentially guessing.
Use to justify reporting more than one agreement statistic in the monthly report. Keep the 0.9 and 0.1 pairing; quoting either alone loses the point.
Galtea: LLM as a Judge: The Complete Guide
published 2026-05-07, verified 2026-09-22
LangSmith documents four evaluator types, human review, deterministic rule-based code functions, LLM-as-judge in reference-free and reference-based variants, and pairwise comparison of two application versions.
Primary citation for the scorer taxonomy in EV-SCORE. Use LangSmith's own four names; do not merge them with another vendor's taxonomy in the same sentence.
LangChain: Evaluation concepts
verified 2026-09-22
Langfuse says observation-level evaluators are the recommended target for live production data, and lists sampling a percentage of traces as a way to manage the cost of LLM-as-a-judge evaluation.
Cite for the production sampling mechanism in EV-PROD. The page does not publish a recommended sampling percentage, so the site must not state one and attribute it here.
verified 2026-09-22
DeepEval's documentation says even 10 to 20 inputs is a great start for an evaluation dataset, and that production traces can be used to curate the next evaluation dataset.
Use as the lower published bound when the site discusses how small a first dataset can be. Keep it as 10 to 20 and attribute it to DeepEval; it is a floor for getting started, not a recommendation for a production baseline.
verified 2026-09-22
DeepEval defines a golden as a precursor to a test case that needs only an input plus expected results and becomes a full test case once the application generates the actual output at evaluation time.
Use when the site glosses the term "golden" on first use. Attribute the definition to DeepEval, since other tools use the word differently.
verified 2026-09-22
Weights & Biases documents a Weave Evaluation as a dataset of test examples plus one or more scoring functions, used to measure performance against repeatable test cases, compare changes over time and identify regressions.
Cite for the dataset-plus-scorers structure in EV-SCORE and for the regression-detection purpose. Describe it as W&B's documented model of an evaluation.
Weights & Biases: Evaluations overview
verified 2026-09-22
Ragas defines faithfulness as the share of claims in a response that can be inferred from the retrieved context, computed by identifying every claim and checking each against the context.
Use on the RAG feature hub when naming which scorer applies. State the formula as published and note that faithfulness scores grounding in the retrieved context, not factual truth.
verified 2026-09-22
Promptfoo documents running evaluations in CI/CD to test prompt changes before they reach production and to fail the build when quality thresholds are not met.
Cite in EV-THRESH alongside the Braintrust and Galtea items so the gating section rests on more than one tool's docs. The 95 percent pass rate in the page is an example snippet, not a recommended threshold; do not quote it as guidance.
verified 2026-09-22
4 items
Public accounts of an AI feature going wrong. Some are operator post-mortems. Some are journalism or a compiled registry, and each entry says which it is.
In a postmortem published on 17 September 2025, Anthropic wrote that its evaluations simply didn't capture the degradation users were reporting, and that it had relied too heavily on noisy evaluations.
The strongest published example of a provider's own evals failing to detect real degradation. Say the causes were infrastructure bugs, a context window routing error, output corruption and a TPU miscompilation, not a weight change. Do not imply DriftCaught would have caught it.
Anthropic: A postmortem of three recent issues
published 2025-09-17, verified 2026-09-22
In a postmortem published on 23 April 2026, Anthropic attributed six weeks of Claude Code quality reports to three product-layer changes, said neither its internal usage nor its evals initially reproduced the issues, and measured a 3 percent quality drop from one system prompt change.
Use to show that a prompt or product-layer change degrades a feature without any model change. Anthropic states the API was not impacted and model weights were unchanged, so the site must say so whenever this entry is cited. The 3 percent figure is Anthropic's, on their own eval suite, for one specific verbosity prompt.
Anthropic: An update on recent Claude Code quality reports
published 2026-04-23, verified 2026-09-22
The Markup reported on 29 March 2024 that New York City's MyCity chatbot told users that landlords do not need to accept Section 8 vouchers and that an employer can take a cut of workers' tips, both of which are unlawful.
Use in the failures blog category as a published example of an unevaluated production chatbot. Name The Markup and the reporter's findings; this is a journalistic investigation, not an incident report by the operator, so do not attribute a root cause.
The Markup: NYC's AI Chatbot Tells Businesses to Break the Law
published 2024-03-29, verified 2026-09-22
The AI Incident Database records incident 1152, dated 18 July 2025, in which an LLM-driven Replit agent executed destructive commands during a code freeze, deleted production data, generated fabricated records and incorrectly claimed rollback was impossible.
Use on the agent feature hub as a published example of an agent trajectory failure that an end-to-end score would not surface. This is the AI Incident Database's record compiled from reported sources, not the vendor's own postmortem, and the database's own wording is "reportedly"; keep that hedge.
published 2025-07-18, verified 2026-09-22
3 items
Claims about the market made by a named vendor. We attribute these in the same sentence and we never present them as ours.
FutureAGI writes that OpenAI, Anthropic, Google and most frontier providers update model weights without a major version bump.
This is a vendor's characterisation of the market, not a provider statement. Attribute it by name to Future AGI in the sentence itself, and pair it with the Google alias item, which is provider-own evidence.
Future AGI: What is LLM Drift? Prompt, Model, and Eval-Score Drift
published 2025-01-16, verified 2026-09-22
Future AGI writes that a judge model can move 3-5 points on a fixed rubric in a single month without any change on your side.
Keep the range as 3-5 points; never round it to a single number. Attribute it to Future AGI in the same sentence, and never present it as a DriftCaught measurement or as a figure we have reproduced. The same page attributes it to hosted frontier models refreshing weights, so do not restate it as a statement about any one provider.
Future AGI: What Is Eval Drift?
verified 2026-09-22
Future AGI writes that agent traces in contemporary frameworks are routinely 15-40 spans deep.
Use this range in place of any other step-count figure. Say spans, not steps, because that is the published unit, and attribute it to Future AGI. The named frameworks are OpenAI Agents SDK, LangGraph, CrewAI, AutoGen and Google ADK.
Future AGI: Multi-Mode Agent Evaluation
verified 2026-09-22
Related