Feature types

What breaks after a model change depends on what the feature does. A summariser and an extractor fail in different ways and need different scorers. Pick the one that matches what you ship.

A feature is one AI capability with its own prompt or pipeline and its own golden dataset. Most products have two or three.

Chat assistantWhat breaks in a multi-turn chat assistant when a provider ships an update, which scorers catch it, what a baseline covers, and which plan fits.13 tracked changes with a note for this type
SummarisationSummarisation has no single correct output, so there is nothing to match. How we score coverage and invention, which scorers apply, and which plan fits.8 tracked changes with a note for this type
ExtractionExtraction is the feature type where rule based code checks earn their place. What breaks after a model change, which scorers apply, and which plan fits.8 tracked changes with a note for this type
RAGRetrieval quality and generation quality are two different failures. How we score each, what faithfulness does and does not measure, and which plan fits.12 tracked changes with a note for this type
AgentAn agent fails in the middle of a run where end-to-end scoring cannot see. What breaks after a model change, which scorers apply, and which plan fits.22 tracked changes with a note for this type
ClassificationClassification has labels, so conventional metrics apply and a judge is often unnecessary. What breaks after a model change and which scorers to report.7 tracked changes with a note for this type
GenerationOpen-ended generation has no correct output to match, so the rubric is the standard. What breaks after a model change, which scorers apply, and which plan fits.12 tracked changes with a note for this type