When you specify a feature with an AI component, the acceptance criterion stops being “given X the system returns Y”. The acceptance criterion becomes an eval: a labelled dataset, a scoring function, and a target score the system must clear before launch and clear again after every model swap or prompt edit. The simple rule: if you cannot write the eval, the feature is not yet specified. This article breaks down the four components of a probabilistic acceptance criterion and how to write them in practice.
What exactly breaks in a traditional spec
A classic specification rests on one assumption: the same input always produces the same output. The entire structure of tests, definitions and exit conditions derives from it. The moment a component of the system is a language model, that assumption collapses, and every tool built on top of it collapses with it.
The most tangible expression of this is the acceptance test. “The user clicks send and receives a summary of the ticket” is not a testable requirement, because there is no single correct answer. Two different summaries can both be excellent, and a third that looks similar can be completely wrong. What is required is a definition of an acceptable behaviour range rather than a single result.
The test that saves months of development: before writing a line of code, try to write the eval. If you cannot define what counts as a good answer and how to measure it, the requirement is still vague. Features that stall in pilot almost always fail at this stage, not at implementation.
The four components of an AI acceptance criterion
| Component | What it defines | Example |
|---|---|---|
| Behaviour description | What the system should do, phrased to admit variability | A ticket summary containing the issue, the action taken and the status |
| Evaluation threshold | A minimum score on a held-out test set never used in development | 90% of summaries include all three elements |
| Safety constraint | What counts as an unacceptable output, regardless of score | No summary ever contains a customer phone number |
| Human review hook | What happens in cases the model cannot resolve | A low confidence score routes the ticket to a human queue |
The difference between the second and third components is fundamental. The threshold is statistical and tolerates isolated failures. The constraint is binary and tolerates none. Without that separation you end up with a feature that passes at 92% success while a data leak hides inside the remaining 8%.
How to build an eval without a research team
An eval is not a project. In the typical case it is three small assets maintained over time:
A dataset. 50 to 150 real cases from the existing system, including the annoying ones. It matters that they are drawn from the field rather than invented, because invented cases are always too convenient. Attach to each case the answer a subject-matter expert would give.
A scoring function. It does not have to be sophisticated. In many cases it is enough to check for the presence of mandatory elements, conformance to a defined structure, or keyword matching. Where judgement is required you can use a model as a judge, provided you manually sample some of the scores to confirm the judge itself is not drifting.
A threshold and a baseline. Measure once before changing anything. Without a baseline you cannot tell whether a prompt change improved or broke things, and most teams discover this too late.
When to run it: an eval is not a one-off check before release. It runs again on every model swap, every prompt edit and every change to a data source feeding the system. That is precisely what separates an eval from a test case, which is why it should be wired into the build process from day one.
Six sections to add to the specification document
| Section | What goes in it |
|---|---|
| Model behaviour specification | What the model should do, in what tone, and what is out of scope |
| Acceptance thresholds | The metrics and minimum scores, including which data they are measured on |
| Failure mode catalogue | The list of ways the feature can fail and what happens in each |
| Behavioural constraints | Binary red lines: what must never appear in the output |
| Evaluation plan | The dataset, the scoring method and the run frequency |
| Data and privacy | What data enters the prompt and at what classification tier |
The last section connects directly to your data classification policy. If you have not defined one yet, the four-tier classification model is the starting point, because without it the specification will define a capability you are not permitted to run.
The failure mode catalogue: the section most worth writing
In a deterministic system a failure mode is an error. In a system with an AI component a failure mode is usually an answer that looks perfectly fine and is not correct. The catalogue therefore has to enumerate failures by what the user experiences, not by what the system reports.
Four are especially common: a confident wrong answer, which requires source citation to be verifiable; a refusal to answer a legitimate question, which requires an alternative route; a partial answer that omits a material condition; and drift over time, where the data source changes and output degrades without anyone touching a line of code. The last is the main reason an eval must run periodically rather than only before release.
What this does to timeline and budget
The practical impact is that effort shifts. In a classic specification most of the time goes into defining screens and flows. In an AI feature a significant share moves to collecting and labelling test cases, and that is subject-matter-expert work rather than developer work. Budget for it explicitly, otherwise it gets squeezed out and you reach release with no way of knowing whether the system works.
A second shift: the feature never “finishes”. It carries an ongoing maintenance cost of running the eval and maintaining the test set. When you calculate viability, that cost belongs in the unit economics from the outset rather than surfacing in the second quarter.
And if the feature answers from your own documents, the specification must cover the retrieval layer too, because that is where most of the real-world accuracy is decided. The full breakdown is in the enterprise knowledge base guide. We support the full process, from specification through release, under systems specification.
Frequently asked questions
What is an eval and how does it differ from a regular acceptance test?
An eval is a labelled dataset together with a scoring function and a minimum threshold. A regular acceptance test checks a single output and returns pass or fail, whereas an eval measures a quality distribution across dozens of cases and reruns on every model, prompt or data source change.
How many test cases do you need to start?
Between 50 and 150 real cases from the existing system is enough for most features. What matters is that they are collected from the field rather than invented, and that they include the hard cases rather than only the convenient ones.
How do you write acceptance criteria for output with no single correct answer?
Define four components: a behaviour description that admits variability, an evaluation threshold on a held-out test set, a binary safety constraint that tolerates no exception, and a routing path to a human for cases the model cannot resolve.
Why separate the evaluation threshold from the safety constraint?
Because the threshold is statistical and tolerates a percentage of failures, while the constraint is binary. Without that separation a feature can pass with a high score while a serious breach, such as exposing personal data, hides inside the failure percentage.
Can you use a model as a judge to score the output?
Yes, and it is a common method where judgement is required. The condition is that you manually sample some of the scores and compare them to human judgement, to confirm the judge itself is not drifting over time.