Run your own suite against your own dataset and compare the result to the last run. Every one of these changes touches something your feature depends on, and a rerun is the only thing that turns that into a fact about your feature.
The default action. Something moved underneath you, so measure it against your dataset.
Meta made muse-voice-transcribe-1.0 available on the Meta Model API for streaming speech to text with speaker diarization. Meta lists the price as $3.00 per 1,000 minutes and offers a WebSocket streaming endpoint and an HTTP batch endpoint.
Summarisation: A transcription step feeds the summariser its input, so the text being summarised changes even though the summarising model did not. Galtea warns against updating ground truth to match model behaviour when the outputs shift.
Extraction: If fields are pulled out of a transcript, a new transcription model changes the input to every fixture. Braintrust tracks every change to a dataset so experiments can pin to specific versions, which is what keeps a before and after run comparable.
Rerun the suiteIf a transcription step feeds a downstream feature, run your existing audio fixtures through this model and compare the transcripts before switching any pipeline.
OpenAI released GPT-6 Astra, which it describes as its most capable model for end to end work, covering reasoning, coding, computer use and tools. It is available in the Responses API and the Chat Completions API.
Agent: A different model plans and calls tools differently. Future AGI describes scoring an agent at four resolutions, end to end, trajectory, step and turn, and writes that a high end to end score can hide bad tool calls, while strong per step scores can still miss that the agent pursued the wrong goal.
Chat assistant: Running the assistant on the new model beside the one you serve is pairwise comparison, one of the four evaluator types LangSmith documents, alongside human review, rule based code functions and LLM as judge.
Summarisation: If a model from this family also runs as your judge, the scoring side is checked against human labels before its verdicts carry weight. Arize writes that a 75 to 90 percent match between judge labels and human labels indicates strong alignment and readiness to scale.
Rerun the suiteAdd gpt-6-astra as a second model in your next scheduled suite run alongside the one you serve today, and read the differences before changing any default.
Google released gemini-3.8-flash as generally available. The entry states it brings improvements for long horizon software engineering.
Agent: Long horizon work is mostly tool calls. Arize Phoenix reconstructs two signals from a trace, tool_path, the ordered tool calls with their arguments, and tool_io, what each tool was called with and what it returned, and warns that a correct looking answer can come from a broken path.
Generation: Generated text has no single correct answer to match against, so the tempting fix when a new model writes differently is to move the expected outputs. Galtea warns against updating ground truth to match model behaviour.
Rerun the suiteQueue gemini-3.8-flash into the next comparison run against the Flash model you use now, and record the result before moving traffic.
OpenAI wrote that a rapid increase in traffic can now return a 429 error with the slow_down code, and that temporary overload can return a 503 error with the server_is_overloaded code.
Rerun the suiteCheck that your client and your eval harness treat these two codes as a signal to back off rather than as failures, then run a batch large enough to exercise that path.
Meta made muse-image-1.0 available to developers on the Meta Model API at $0.01 per image. Meta describes anchored composition for consistency across a series of images and reasoning driven edits.
Generation: There is no single correct output to match a generation prompt against, so a fixed prompt set and the recorded outputs carry the comparison. Galtea warns against updating ground truth to match model behaviour.
Rerun the suiteIf image generation sits inside a product flow, run your standing prompt set against this model and keep the outputs as a baseline for the next version.
OpenAI reduced GPT-5.6 Sol pricing to $4 per million input tokens and $20 per million output tokens. The entry states the reduced price is promotional through November 21.
Rerun the suiteUpdate the per token rates in your cost model and dashboards, and put the stated promotional end date in the budget calendar.
Google released gemini-3.7-flash. The entry states it brings substantial improvements across software engineering.
Chat assistant: Putting the new Flash model beside the one you pin is pairwise comparison of two application versions, one of the four evaluator types LangSmith documents, alongside human review, rule based code functions and LLM as judge.
Rerun the suiteAdd gemini-3.7-flash to the comparison run you already schedule for Flash tier models, and note where its answers differ from the version you pin.
Meta released Muse-Glimmer-30B, a 30 billion parameter open weight model for local agent workloads, under the Apache 2.0 licence with a 128K token default context window. Meta lists quantized builds and support for vLLM, llama.cpp, ExecuTorch, Ollama and LM Studio.
Agent: An open weight model is pinned by you, so the swap happens when you upgrade rather than when the provider ships. Future AGI writes that agent traces in contemporary frameworks are routinely 15-40 spans deep, so the comparison reads the trace and not the final answer alone.
Rerun the suiteIf you self host a Llama 4 model, run your current suite against Muse-Glimmer-30B on the same hardware and record both quality and throughput before planning a move.
Meta released muse-spark-1.2 with a 1M token context window, priced at $1.25 per million input tokens, $0.15 per million cached input tokens and $4.25 per million output tokens. Meta states the model was co-trained with its harness and trained on whole repository generation and other long running tasks.
Agent: Whole repository work and other long running tasks are scored end to end by default. Arize Phoenix writes that end to end evaluation only inspects the two endpoints, the input and the final output, and that reasoning, tool selection and the decision path are where the interesting failures sit.
Generation: Running 1.2 and 1.1 in the same pass is pairwise comparison of two application versions, one of the four evaluator types LangSmith documents.
Rerun the suiteRun your coding and long context cases against muse-spark-1.2 and muse-spark-1.1 in the same pass, and note that the contributor tier is rate limited by tokens in a rolling window rather than by request count.
OpenAI states that Fast mode now supports long context requests exceeding 272K tokens for the GPT-5.6 models.
RAG: Dropping the chunking you added for the old ceiling changes what sits in the context window. Ragas defines faithfulness as the share of claims in a response that can be inferred from the retrieved context, so the score moves with the context as well as with the model.
Summarisation: Summarising a whole document rather than a set of chunks changes the expected output for every case. Galtea warns against updating ground truth to match model behaviour.
Rerun the suiteIf you split long inputs to stay under the old ceiling, run a long document case end to end before removing that chunking.
OpenAI introduced Fast mode, which it describes as up to 2.5 times faster at twice the price. The entry also states that GPT-5.6 Luna costs 80 percent less and GPT-5.6 Terra 20 percent less.
Rerun the suitePrice the three GPT-5.6 tiers against your current spend per request, then run the cheaper tier through your suite before moving any traffic to it.
Meta announced Muse Spark 1.1, a multimodal reasoning model built for agentic tasks, with what it calls major gains in tool and computer use, coding and multimodal understanding. Meta states it has a 1 million token context window and launched a public preview of the Meta Model API for developer access.
Agent: Changes in tool and computer use land in the middle of the trace. Arize Phoenix reconstructs tool_path, the ordered tool calls with their arguments, and tool_io, what each tool was called with and what it returned, and warns that a correct looking answer can come from a broken path.
Rerun the suiteIf you track Meta models, open an account on the Meta Model API and run your existing suite once so later versions have a baseline to be compared against.
Meta put muse-spark-1.1 in public preview on the Meta Model API for United States developers, priced pay as you go at $1.25 per million input tokens and $4.25 per million output tokens. Meta lists end to end agentic workflows, advanced coding and native multimodal perception as the areas of improvement.
Rerun the suiteAdd the published input and output rates to your cost model, and note that the preview is limited to United States developers when you plan access.
Google launched public preview support for the Computer Use tool in Gemini 3.5 Flash.
Agent: Future AGI describes scoring an agent at four resolutions, end to end, trajectory, step and turn, writes that a high end to end score can hide bad tool calls, and warns that strong per step scores can still miss that the agent pursued the wrong goal.
Rerun the suiteRun your agent cases with the Computer Use tool enabled on gemini-3.5-flash and keep the recorded tool calls, not only the final answers.
OpenAI wrote that for organizations without ZDR, its zero data retention setting, prompt_cache_retention now defaults to 24h instead of in_memory.
Rerun the suiteCheck whether your requests set prompt_cache_retention explicitly, and decide whether the new default is the one you want before your next run.
Anthropic launched claude-opus-4-8 with a 1M token context window by default on the Claude API, Amazon Bedrock, Google Cloud and Microsoft Foundry. The notes state that the effort parameter defaults to high, that the minimum cacheable prompt length was reduced to 1,024 tokens, that adaptive thinking triggers reasoning only when needed, and that mid conversation system messages are available without a beta header.
Agent: An effort parameter that defaults to high changes how much happens between the request and the answer. Arize Phoenix writes that end to end evaluation only inspects the two endpoints, the input and the final output, and that reasoning, tool selection and the decision path are where the interesting failures sit.
Chat assistant: Mid conversation system messages change the shape of a multi turn request. LangSmith documents four evaluator types for comparing versions, human review, rule based code functions, LLM as judge in reference free and reference based variants, and pairwise comparison.
Rerun the suiteRun your suite once at the default effort of high before you override it, and re-check cache hit rates now that shorter prompts qualify for caching.
Google released gemini-3.5-flash as generally available. The same entry launches Managed Agents in public preview and an antigravity-preview-05-2026 general purpose managed agent.
Chat assistant: Moving a preview pin onto a generally available id is the direction Google's own documentation points, saying most production apps should use a specific stable model and hedging that stable models usually do not change.
Agent: A managed agent in public preview carries a shorter clock than a stable model. Google says Gemini preview models may be used in production but will be deprecated with at least 2 weeks' notice.
Rerun the suiteMove any preview Flash pin onto the generally available id and run the suite once to see whether the outputs shifted.
OpenAI released GPT-5.5 as a frontier model with a 1M token context window, structured outputs and extended prompt caching, alongside gpt-5.5-pro.
Extraction: Structured outputs decide whether extracted fields validate against your schema. DeepEval defines a golden as an input plus the expected result, which becomes a full test case once the application produces the actual output, so the same goldens rerun against the new model.
RAG: A larger window invites more retrieved text into the prompt. Ragas defines faithfulness as the share of claims in a response that can be inferred from the retrieved context, and it scores grounding in that context rather than factual truth.
Rerun the suiteRun your structured output cases against gpt-5.5 and check that responses still validate against your schemas before changing the model you serve.
Anthropic launched claude-opus-4-7 at $5 and $25 per million tokens, the same price as Claude Opus 4.6. The notes state that task budgets launched in beta, that high resolution image input is supported up to 2576 pixels on the long edge, and that an xhigh effort level was added.
Agent: Task budgets and a new effort level change how much the agent does before it answers. Future AGI writes that agent traces in contemporary frameworks are routinely 15-40 spans deep, so the comparison reads the trace rather than the final answer alone.
Extraction: Higher resolution image input means page and screenshot fixtures may be re-encoded, which changes the inputs themselves and not only the model. Braintrust tracks every change to a dataset so experiments can pin to specific versions.
Rerun the suiteRun the suite against claude-opus-4-7 at your current effort level first, then once at xhigh, and record what the extra effort changes before you make it the default.
Google released the Built-in Tools and Function Calling Combination feature, which makes it possible to use Gemini's built-in tools alongside custom function calling tools in a single API call. The same changelog entry says Grounding with Google Maps is now supported for Gemini 3 models going forward.
Agent: Mixing built in tools with custom functions changes which tool is chosen and in what order. Arize Phoenix describes reading tool_path, the ordered tool calls with their arguments, and tool_io, what each tool was called with and what it returned, and warns that a correct looking answer can come from a broken path.
Rerun the suiteIf you combine a built in tool with your own functions in one call, record the tool calls for a set of cases before and after, so a changed selection order is visible.
OpenAI released GPT-5.4 mini and GPT-5.4 nano to the Chat Completions and Responses APIs.
Classification: Weights & Biases documents an evaluation as a dataset of test examples plus one or more scoring functions, used to measure performance against repeatable test cases, compare changes over time and identify regressions. That is the structure a move to a smaller id is judged on.
Rerun the suiteIf you are considering a cheaper id for routing or tagging, run the same labelled set through gpt-5.4-mini and gpt-5.4-nano and compare it against the model you use today.
Anthropic took the 1M token context window out of beta for Claude Opus 4.6 and Claude Sonnet 4.6, at standard pricing and with no beta header required. The notes state that the dedicated 1M rate limits were removed and that the media limit was raised to 600 images or PDF pages per request.
RAG: A longer window changes how much retrieved text you put in front of the model. Ragas defines faithfulness as the share of claims in a response that can be inferred from the retrieved context, so the score follows the context you assemble.
Summarisation: Feeding a whole document instead of chunks changes the input to every case, which makes the old run and the new run two different things unless the inputs are tracked. Braintrust tracks every change to a dataset so experiments can pin to specific versions.
Rerun the suiteRemove the 1M beta header from your clients and rerun any long context case, since the separate rate limit pool that used to back those requests is gone.
OpenAI wrote that it updated its image encoder to fix a small bug with input_image inputs in gpt-5.4. OpenAI describes it as a bug fix and publishes no new model id alongside it.
Rerun the suiteIf a feature sends images to gpt-5.4, run your image cases again after this date and keep both sets of outputs, because the id you call did not change.
OpenAI released GPT-5.4 as a frontier model with a 1M token context window, tool search, computer use and compaction, alongside gpt-5.4-pro.
Agent: Tool search and computer use add steps that the final answer never shows. Arize Phoenix reconstructs tool_path, the ordered tool calls with their arguments, and tool_io, what each tool was called with and what it returned, and warns that a correct looking answer can come from a broken path.
RAG: Compaction changes what the model still has in front of it when it answers. Ragas defines faithfulness as the share of claims in a response that can be inferred from the retrieved context, so a compacted context is a different thing to be faithful to.
Rerun the suiteRun one long input case against gpt-5.4 before you rely on the larger window, and check how compaction changes what the model still has in front of it.
OpenAI released gpt-5.3-codex to the Responses API. The same entry added the phase parameter, which OpenAI says labels an assistant message as intermediate commentary or final answer.
Agent: Future AGI describes scoring an agent at four resolutions, end to end, trajectory, step and turn, writes that a high end to end score can hide bad tool calls, and warns that strong per step scores can still miss that the agent pursued the wrong goal. A label that separates intermediate commentary from the final answer is read at those resolutions.
Rerun the suiteRun your agent cases against gpt-5.3-codex, and decide whether your harness should record the phase label so intermediate messages stop being scored as answers.
Google released gemini-3.1-pro-preview and launched a gemini-3.1-pro-preview-customtools variant.
Agent: A custom tools variant changes how the agent reaches its answer, not only what it answers. Future AGI describes scoring an agent at four resolutions, end to end, trajectory, step and turn, and writes that a high end to end score can hide bad tool calls, while strong per step scores can still miss that the agent pursued the wrong goal.
Rerun the suiteRun the Pro tier comparison against gemini-3.1-pro-preview, and keep the custom tools variant in a separate run so tool behaviour differences stay visible.
Anthropic launched claude-sonnet-4-6 at $2 and $10 per million tokens with a 1M token context window in beta and extended thinking support. The notes state that web search and web fetch are free when used with code execution, and that web search, web fetch, programmatic tool calling, code execution, tool search, tool use examples and the memory tool are available without a beta header.
Agent: Tools that no longer need a beta header are tools a request can now reach. Arize Phoenix reconstructs tool_path, the ordered tool calls with their arguments, and tool_io, what each tool was called with and what it returned, and warns that a correct looking answer can come from a broken path.
RAG: Where web search or web fetch supplies the documents, the retrieved context changes with the tool path. Ragas defines faithfulness as the share of claims in a response that can be inferred from the retrieved context, and it scores grounding in that context, not factual truth.
Rerun the suiteRemove the beta headers you no longer need from tool requests and rerun the tool using cases, since header changes can alter which tool path a request takes.
OpenAI launched server side compaction in the Responses API, launched Skills support across local and hosted execution, and released the Hosted Shell tool with support for networking in containers.
Agent: Arize Phoenix writes that end to end evaluation only inspects the two endpoints, the input and the final output, and that the interesting failures happen in the middle where they cannot be seen, naming reasoning, tool selection and decision path. Compaction and a new hosted tool both sit in that middle.
Rerun the suiteIf you turn on server side compaction or adopt the Hosted Shell tool, run a long session case with the spans recorded rather than only the final answer.
Google launched support for the Computer Use tool in gemini-3-pro-preview and gemini-3-flash-preview.
Agent: Arize Phoenix describes reconstructing tool_path, the ordered tool calls with their arguments, and tool_io, what each tool was called with and what it returned, and warns that a correct looking answer can come from a broken path.
Rerun the suiteBoth ids are preview models, so keep the rerun cycle short for anything built on this tool, and record which preview id each run used.
OpenAI released gpt-5.2-codex to the Responses API and describes it as optimized for agentic coding tasks in Codex.
Agent: Arize Phoenix describes reconstructing tool_path, the ordered tool calls with their arguments, and tool_io, what each tool was called with and what it returned, and warns that a correct looking answer can come from a broken path. Those are the signals a coding agent is read on.
Rerun the suiteIf a coding or agent feature sits on a pinned Codex id, run your trajectory cases against gpt-5.2-codex before you point anything at it.
Google renamed the total_reasoning_tokens field to total_thought_tokens, saying the new name better aligns with the concept of thoughts in thinking models. The change affects v1beta.
Rerun the suiteUpdate anything that reads total_reasoning_tokens from a v1beta response, including cost dashboards and eval harness logging, then run a batch and check the field is populated.
Google launched gemini-3-flash-preview. The entry describes frontier class performance and upgraded reasoning capabilities.
Summarisation: A summariser pinned to a preview id runs on a shorter clock. Google says Gemini preview models may be used in production but will be deprecated with at least 2 weeks' notice.
Rerun the suiteAdd gemini-3-flash-preview to your Flash tier comparison run, and keep the preview id out of production pins until a stable id ships.
OpenAI released GPT-5.2 as its flagship model, stating improvements in reasoning, vision, code generation and spreadsheet understanding, alongside gpt-5.2-chat-latest. The same day OpenAI released client side compaction for long running conversations.
Extraction: Vision and spreadsheet inputs are where extracted fields come from. DeepEval defines a golden as an input plus the expected result, which becomes a full test case once the application produces the actual output, so the document fixtures rerun unchanged against the new model.
Generation: Running the new flagship beside the snapshot you pin is pairwise comparison of two application versions, one of the four evaluator types LangSmith documents.
Rerun the suiteRun your suite against gpt-5.2 alongside the GPT-5.1 snapshot you pin, and pay attention to the vision and document cases the entry calls out.
Anthropic launched Claude Opus 4.5, which the release notes describe as its most intelligent model, combining maximum capability with practical performance. The release notes name the family alias claude-opus-4-5. Anthropic's model deprecations table lists the dated snapshot claude-opus-4-5-20251101 as Active, with retirement not sooner than 24 November 2026.
Rerun the suiteA new model is an option rather than an event, unless an alias you already call now resolves to it. Run your own dataset against the model you use today and against this one before deciding to move.
Google launched gemini-3-pro-preview. The entry describes state of the art reasoning and multimodal capabilities.
Chat assistant: A first baseline on a preview id is a baseline with a short life. Google says Gemini preview models may be used in production but will be deprecated with at least 2 weeks' notice.
Rerun the suiteSet up a first baseline run against gemini-3-pro-preview so later Gemini 3 versions have something to be compared with.
Anthropic launched Claude Haiku 4.5, which the release notes describe as its fastest and most intelligent Haiku model, with near-frontier performance. The release notes name the family alias claude-haiku-4-5. Anthropic's model deprecations table lists the dated snapshot claude-haiku-4-5-20251001 as Active, with retirement not sooner than 15 October 2026.
Rerun the suiteIf you are considering this model to cut cost or latency, the comparison that matters is your own dataset scored on both models, not a published summary of either.
OpenAI released GPT-5 Pro, which it describes as a version of GPT-5 that uses more compute to think harder. The same DevDay entry launched the v1/videos API with the Sora 2 and Sora 2 Pro models, and released GPT-Realtime mini, gpt-audio-mini and gpt-image-1-mini for cost efficient speech and image work. It also launched Agent Builder, ChatKit and Trace Evals.
Generation: A release that adds generation models gives a team a new option rather than a forced change. Galtea warns against updating ground truth to match model behaviour, so the outputs you already recorded stay the comparison point when a new id is tried.
Rerun the suiteNothing moves until you adopt one of these ids. If you plan to try gpt-5-pro, run your standing prompt set against it first and keep the current model's outputs beside it.
Anthropic launched claude-sonnet-4-5-20250929, which it describes as its best model for complex agents and coding. The notes state that a new stop reason model_context_window_exceeded was added, that the memory tool launched in beta and that context editing launched in beta.
RAG: Context editing and a new stop reason change what actually reaches the model on a long input. Ragas defines faithfulness as the share of claims in a response that can be inferred from the retrieved context, so the score follows whatever survived.
Agent: A memory tool adds calls that the final answer never shows. Arize Phoenix reconstructs tool_path, the ordered tool calls with their arguments, and tool_io, what each tool was called with and what it returned.
Rerun the suiteAdd handling for the model_context_window_exceeded stop reason so long inputs fail visibly, and rerun the long context cases that previously truncated.
Change-log entries restate provider announcements with sources and dates. Impact notes cite public practitioner reports. The site publishes no measurements of its own on any model.
Questions about this action
Who decides the recommended action?
We do, from what the provider announced, and it is one of exactly four: rerun, re-pin, re-prompt or migrate. It is a statement about what a team does this week, not a prediction about your feature.
Does an action mean something is wrong with my feature?
No. It means something your feature depends on moved. Whether anything about your output changed is a question only a rerun against your own dataset can answer.
What happens if we are on a plan?
A tracked provider update triggers a rerun of your suite within 2 business days of the provider's announcement, and you get a written verdict. That is one of the six written guarantees on every plan.