Testing an AI agent is neither software testing nor model benchmarking. An agent runs a path that changes between runs, so the only meaningful measurement is a distribution: same task, same conditions, twenty runs — how many finished correctly. The arithmetic is simple and brutal: an agent that is 95% accurate at every step completes only about 60% of ten-step tasks correctly. A working eval setup has four parts: a case set drawn from the real world, measurement at four levels, a regression run on every change, and production monitoring.
Why ordinary software testing does not work here
Conventional software testing assumes determinism: the same input produces the same output, so covering the paths is enough. In an agent that assumption does not hold. The same input can produce a different path, a different tool order and a different result — not because of a bug, but because of how the model operates.
So the testing question changes. Instead of pass or fail, the question is how often it succeeds, and how far the worst failure sits from the average. The teams that get burned are the ones who saw a successful demo, concluded the system works, and discovered in production that the demo was the best run out of five.
The success metric most commonly quoted in the industry measures one success under optimal conditions. In production what matters is stability — same task, many runs. The gap between those two numbers is the gap between a demo and a system.
The arithmetic that explains almost everything
An agent performs a task as a chain of steps. If each step is independent, the success rate of the task is the product of the probabilities. That looks trivial until you put numbers in.
| Accuracy per single step | 5-step task | 10 steps | 20 steps |
|---|---|---|---|
| 90% | 59% | 35% | 12% |
| 95% | 77% | 60% | 36% |
| 98% | 90% | 82% | 67% |
| 99% | 95% | 90% | 82% |
Two operational conclusions follow. First: shortening the task is worth more than improving the model. Going from twenty steps to ten at 95% accuracy nearly doubles the success rate, from 36% to 60%. Second: long chains require a per-step accuracy that is very hard to reach, so long tasks demand checkpoints and mid-run validation rather than one continuous loop.
This is also why unusually long tasks still fail most of the time even with the best models available. Breaking work into sub-tasks with intermediate validation is not an engineering compromise — it is the only route the mathematics allows.
How to build a case set that actually teaches you something
A test set written by the person who built the agent will fail in production, because it reflects what the developer imagined. The right set comes from three sources.
| Case type | Where it comes from | How much |
|---|---|---|
| Happy path | The last twenty requests handled manually | ~40% of the set |
| Real edge cases | Complaints, refunds, cases that required escalation | ~40% of the set |
| Adversarial cases | Injection attempts, missing input, contradictions, out-of-scope requests | ~20% of the set |
Every case needs a success condition defined in advance. Not a subjective score but a checkable condition: the field was updated to the correct value, the email went to the correct address, the total matches the manual calculation, or the agent refused to act and escalated. A case without a success condition is not a test case, it is an impression.
A reasonable starting size: 40 to 80 cases, each run five to ten times. That sounds like a lot, but it is a few minutes of runtime and a few dollars in cost — against one customer who received a wrong answer.
Four levels of measurement
Measuring the outcome alone hides the cause. A useful setup measures four levels in parallel.
| Level | What is measured | What it reveals |
|---|---|---|
| Outcome | Whether the success condition was met | Whether the system clears the business bar |
| Path | The order of tools chosen and the number of steps | Where the agent starts drifting, before the outcome breaks |
| Tool | Rate of calls with a wrong argument or an unsuitable tool | Which tool descriptions need fixing |
| Cost | Tokens, calls and time per task | Whether the solution is economical at scale |
The path level is the most important one for debugging. An agent error usually surfaces far from where it was created, so without a full log of every call and result there is no way to find the step that started the drift. The structural background is explained in Anatomy of a Business AI Agent.
Regression: what breaks an agent you never touched
An agent is one of the few systems that can break while nobody changed a line inside it. There are three main causes.
Model update. The provider ships a new version and behaviour shifts — usually better on average and worse on specific cases that were calibrated to the previous version. Every model upgrade therefore requires a full run of the set before switching.
Tool schema change. A field added, a name changed, a new default value. The agent will not crash; it will keep calling the tool the way that was correct yesterday.
Data drift. The shape of incoming requests changes over time, new products arrive, customer phrasing shifts. A one-year-old test set is testing a world that no longer exists.
A simple working rule: the set runs automatically on every change to the prompt, the tools or the model, and is refreshed quarterly from real traffic.
Production monitoring — the only thing that measures the truth
The set measures what you thought to test. Production measures what happens. Four metrics are enough to catch almost any problem in time.
- Completion rate. The share of tasks that finished without human intervention and without failure.
- Escalation rate. How often the agent stopped and asked for a human. A sudden rise signals a change in input; a sudden drop signals overconfidence.
- Correction rate. How often a human changed what the agent proposed. This is the metric closest to real quality.
- Cost per successful task. Not cost per call. Tasks that fail and get re-run are the real cost — the financial side is detailed in the guide to reducing inference cost.
The correction rate is also the clearest sign of whether the approval point sits in the right place. If almost nobody corrects anything, approval has become a rubber stamp — a subject covered in the human-in-the-loop guide.
A checklist before the agent touches a customer
- Every case in the set has an unambiguous success condition written before the run.
- Every case runs at least five times, and the reported metric is stability rather than the best run.
- The set includes adversarial cases, including injection attempts through every input channel — see the permissions guide.
- A full log of tool calls and results exists for every run, stored and searchable.
- A numeric business bar for going live has been defined, rather than a feeling of good enough.
- A fallback scenario has been defined: what happens when the agent fails, and who receives the task.
Frequently Asked Questions
What are evals for an AI agent?
A test setup that measures agent behaviour across a predefined case set, each case over several runs, and reports the distribution of results rather than a single success. It covers the outcome, the path, tool usage and cost.
How many test cases do you need?
A starting set of 40 to 80 cases covers most of the behaviour of a single-task agent, provided it is split between happy path, real edge cases and adversarial cases. More important than the number of cases is the number of runs per case, which should be at least five.
Why does an agent that worked in the demo fail in production?
Because the demo measured one run under convenient conditions. In production the task is longer, the input is dirtier, and errors compound across steps. An agent with 95% per-step accuracy completes only about 60% of ten-step tasks correctly.
Can you use a model to judge the agent?
Yes, but only as a complementary layer. Automated judging suits qualitative measures such as tone or answer completeness, not success conditions that can be checked deterministically. Any case that can be verified by comparing values should be verified that way.
When do you run the test set?
On every change to the prompt, the tools or the model version, plus a periodic refresh of the set from real traffic. An agent can break even when nobody touched it, mainly following a model update or a tool schema change.
Continue reading
Agent Observability: The Unit of Measurement Is the Step, Not the Answer – Evals gate the release. Observability tells you what happened after it. The two are not interchangeable.