Stacking three levers — routing between models, caching, and batch processing — cuts an inference bill by 60 to 80 percent with no change to the user experience. Prompt caching alone prices repeated tokens at roughly 10 percent of the base input rate, meaning up to a 90 percent discount on the fixed portion, and it also improves time to first token by 13 to 31 percent. Batch processing removes about another 50 percent from anything that does not need an immediate answer. Stacked together, the bill lands near a quarter of list price.
Where the money actually burns
Before optimising, you need to know what is expensive. Three structural facts explain most of the bill, and they hold across essentially every provider.
Output tokens cost five to eight times what input tokens cost. On GPT-5.2, for instance, the rate is 1.75 dollars per million input tokens against 14.00 dollars per million output tokens. The practical consequence is that shortening the answer is worth far more than shortening the prompt, and most teams invest in exactly the opposite.
The gap between the expensive model and the cheap one is an order of magnitude, not a percentage. The cheap tier prices around 0.10 dollars per million input tokens. When the flagship costs fifteen times that, every request sent to it unnecessarily is waste you cannot recover later.
The fixed portion of the prompt repeats on every request. System instructions, the schema, the examples, the business rules — usually thousands of tokens sent again and again at full price each time. This is the portion caching almost entirely eliminates.
Measure before you optimise. Break the monthly bill down by request type, and for each type record volume, average input tokens, average output tokens, the model in use, and how many calls one task consumes. In most organisations two request types generate over 70 percent of the bill, and all the work should start there. The full cost breakdown is in what an AI feature really costs.
Six levers, ranked by return on effort
| Lever | Typical saving | Effort | Quality risk |
|---|---|---|---|
| Prompt caching | 45 to 80 percent on the repeated portion | Low. Reorder the prompt | None |
| Model routing | 40 to 85 percent | Medium. Needs a classifier and an eval set | Medium without measurement |
| Batch processing | About 50 percent on non-urgent work | Low to medium. Needs a queue | None |
| Output shortening and structured schema | 20 to 40 percent | Low | Low |
| Semantic cache | Part of 47 percent when combined with routing | Medium. Needs embeddings and a threshold | Medium. A stale answer to a new situation |
| Model distillation | Up to about 90 percent on the trained task | High. Needs data and maintenance | Low on task, high outside it |
Note the effort column. The first three rows deliver most of the saving in weeks, not quarters. Distillation and quantisation are real steps, but they belong to a later stage, once volume justifies them.
Model routing: the single highest-return move
The assumption that every request needs the strongest model is the most expensive assumption in the architecture. In practice, most traffic in any product is simple: classification, field extraction, rewriting, questions that have been answered a thousand times. Those run perfectly well on the cheap tier.
Complexity-based routing removes 40 to 85 percent of the bill, and combined with a semantic cache has been measured at a 47 percent spend reduction even in systems that already looked optimised. The safe way to implement it has three steps.
First, build an eval set before you touch routing. One to two hundred real examples from live traffic, with an agreed correct answer. Without it there is no way to know whether routing broke something, and you will find out from customers.
Second, route only the easy work. Start with one clear category of simple requests, move it to the cheap model, and compare against the eval set. If the gap is negligible, expand to the next category.
Third, add automatic escalation. When the cheap model returns low confidence or an answer that fails schema validation, send the request to the strong model. Even at a 15 percent escalation rate, the saving on the other 85 percent covers it comfortably.
Caching: two entirely different layers
The word cache describes two things here that get confused with each other, and they solve different problems.
Prompt caching is the provider mechanism. It recognises that the prefix of a request matches a previous prefix and prices it at roughly 10 percent of the input rate. The condition is that the fixed portion appears at the start of the prompt and the variable portion at the end — the reverse order voids the cache entirely. This is almost always the first improvement worth making, because it takes an hour and also improves time to first token by 13 to 31 percent.
Semantic caching is your mechanism. A new question is converted to an embedding, compared against previous questions, and if similarity clears a defined threshold the stored answer is returned without calling the model at all. It is powerful in products with repeated questions, but it is also the only item on this list that can return a wrong answer with full confidence.
Two rules for semantic caching. First, never cache anything that depends on time, price, stock or the user permissions — those are exactly the cases where a stale answer causes damage. Second, set a high similarity threshold and a short expiry, and measure how often the cache returned an answer the user rejected. If that rate rises above low single digits, the threshold is too loose.
Batch, output shortening and architecture
The three remaining levers are less glamorous and therefore less implemented, even though the first is the easiest discount in the industry.
Batch processing. Every major provider offers roughly a 50 percent discount on requests willing to wait. Nightly summaries, record enrichment, historical labelling, description generation — none of these need an answer within a second. The only obstacle is that you need a queue, and most teams simply never built one. It is days of work that returns half the cost of everything routed through it.
Output shortening. Because output costs five to eight times more, capping answer length is the most direct lever available. Requesting output in a structured schema instead of free text saves between 20 and 40 percent and produces answers that are easier to validate. If your system generates long reasoning that nobody reads, you are paying full price for it.
Quantisation and distillation, when volume justifies them. Running a model at INT8 or INT4 precision cuts about 50 percent of the cost while retaining 95 to 99 percent of accuracy. Distilling a small model for a specific task reaches up to about 90 percent saving on that task. Both require infrastructure and ongoing maintenance, so they are right when volume is stable and large — not before. That trade-off is the same one described in build or buy AI capability.
The order of work, and what not to do
Order matters because each lever changes the baseline for the next. Measure, then the three easy ones, then the rest.
| Step | What you do | When to stop |
|---|---|---|
| 1. Measure | Break the bill down by request type, model and volume | Once you know the two most expensive request types |
| 2. Prompt caching | Reorder: fixed content first, variable content last | When the cache hit rate is stable and reported |
| 3. Schema and output length | Structured output, a length ceiling, no reasoning nobody consumes | When eval-set quality has not moved |
| 4. Batch | Move everything non-urgent to a queue | When what remains synchronous is genuinely interactive |
| 5. Routing | One category at a time, with automatic escalation | When escalations settle below 20 percent |
| 6. Distillation or quantisation | Only for a stable, high-volume task | When monthly saving exceeds maintenance cost |
And three things not to do. Do not trade quality for cost without an eval set showing exactly what degraded — a saving that generates one more support contact is a loss. Do not optimise before you have volume, because optimising small traffic spends expensive engineering time for a trivial return. And do not hide the saving: if cost per unit of work fell 60 percent, that changes your pricing and it changes the pilot ROI, and it is usually what turns a marginal pilot into a justified one. We covered the link between cost structure and price in pricing a model-backed product.
Building a cost model and a cost-reduction plan is part of the financial and growth planning work we do with clients.
Frequently asked questions
How much can you reduce inference cost?
Stacking model routing, caching and batch processing cuts 60 to 80 percent of the bill with no degradation to the user experience. Prompt caching alone prices repeated tokens at roughly 10 percent of the input rate, batch adds about a 50 percent discount on non-urgent work, and routing removes a further 40 to 85 percent. Stacked together the bill lands near a quarter of list price.
What is the first step to reducing cost?
Measurement, followed by prompt caching. Break the bill down by request type, model and volume — in most organisations two request types produce over 70 percent of the cost. Then reorder the prompt so the fixed portion comes first and the variable portion last. That is an hour of work returning 45 to 80 percent on the repeated portion, and it improves time to first token by 13 to 31 percent as well.
Does model routing hurt quality?
Not if you build an eval set before routing. Prepare one to two hundred real examples with an agreed correct answer, route one category of simple requests at a time, and compare against the set. Add automatic escalation to the strong model when confidence is low or the answer fails schema validation. Even at a 15 percent escalation rate, the saving on the rest covers it easily.
When is it worth distilling a dedicated model?
Only when the task is stable, volume is high, and the monthly saving exceeds the maintenance cost. Distillation reaches up to about 90 percent saving on the trained task, and quantisation to INT8 or INT4 cuts about 50 percent while retaining 95 to 99 percent of accuracy. Both require data, infrastructure and continuous upkeep, so they are a late step rather than a first one.
Why do output tokens matter more than input tokens?
Because they cost five to eight times more. On GPT-5.2 the rate is 1.75 dollars per million input tokens against 14.00 dollars per million output tokens. Capping answer length and moving to structured schema output therefore saves 20 to 40 percent of cost, and usually makes answers easier to validate too. If the system produces long reasoning that nobody reads, you are paying full rate for it.
Continue reading
Agent Observability: The Unit of Measurement Is the Step, Not the Answer – You cannot cut inference cost you cannot attribute to a specific step.