Token cost is one line out of seven in the total cost of ownership of an LLM feature. A full account includes input and output tokens, retries, retrieval and embeddings, an evaluation layer, monitoring, human oversight and infrastructure. The published API price usually explains less than half of what the feature actually costs in production, which is why traditional SaaS gross margins of 80-90% compress to 50-60% once model calls enter the product. The right question is not what a million tokens cost, but what one unit of work a customer will pay for costs.
The seven cost lines of a model-backed feature
Most teams price an AI feature straight off the vendor’s rate card: so many dollars per million input tokens, so many for output. That is one line. Production surfaces six more, and several of them are fixed rather than volume-linked, which means they weigh most heavily at the start.
| Cost line | What it covers | Behaviour |
|---|---|---|
| Input and output tokens | The prompt, the retrieved context, and the model’s answer | Variable, linear with volume |
| Retries and failures | Timeouts, rate limits, output rejected by validation and re-sent | Variable, spikes under load |
| Retrieval and embeddings | Embedding generation, vector storage, retrieval queries, re-indexing | Mixed: indexing fixed, querying variable |
| Evaluation layer | Test sets, eval runs before every prompt or model change | Fixed per development cycle |
| Monitoring and logging | Input and output capture, quality metrics, alerting, retention | Fixed with a volume component |
| Human oversight | Sample review, escalation handling, edge-case labelling | Variable in person-hours |
| Infrastructure and integration | Provider abstraction layer, queues, caching, security and permissions | Fixed, shrinks proportionally as you grow |
Why the published token price misleads
Three systematic gaps separate the rate card from the invoice.
Output is far more expensive than input. With most providers an output token is priced five to eight times an input token. A feature that generates long answers pays for that length twice over, which makes capping output length the cheapest saving available on any given day.
Context grows quietly. The moment retrieval enters the picture, every call carries document chunks, conversation history and system instructions. A prompt that looks like three lines in code is several thousand tokens in flight, and that difference multiplies across every call.
Retries never appear in the plan. A call that fails validation, formatting or a timeout is sent again and billed twice. Under real load the retry rate is usually the difference between the forecast and the invoice.
The formula: cost per unit of work
Instead of tracking total token consumption, define one unit of work with business meaning: a ticket resolved, a document summarised, a lead enriched, a row classified. The arithmetic then looks like this:
| Component | How to compute it | Note |
|---|---|---|
| Average call cost | (input tokens × input price) + (output tokens × output price) | Measure it, do not estimate it |
| Calls per unit | Including retrieval, routing, classification and checking steps | A typical agent makes 3-8 calls per unit |
| Retry multiplier | 1 divided by the first-attempt success rate | 85% success equals a 1.18 multiplier |
| Human oversight cost | Escalation rate × loaded hourly cost × handling time | Usually the heaviest line |
| Fixed cost allocation | Total fixed cost divided by monthly units | Falls as volume rises |
Benchmarks published through 2026 in customer service put an AI-resolved ticket at roughly half a dollar to two dollars at the unit level, against six to twelve dollars for a human-handled contact. The gap looks decisive, but it narrows considerably once oversight, evaluation and infrastructure are added back, and some published figures put the fully loaded cost of an automated resolution nearer five dollars. The distance between those two readings is exactly the distance between token cost and cost of ownership.
The three levers that actually move the bill
Before changing architecture, there are three discounts providers already offer that most teams do not fully exploit.
| Lever | Order of magnitude | When it fits |
|---|---|---|
| Prompt caching | Cache reads priced around a tenth of the standard input rate | Stable system instructions, recurring base documents |
| Batch processing | Roughly half price | Anything that does not need a real-time answer |
| Model selection per task | Tens of times difference between model tiers | Classification, extraction and routing do not need a frontier model |
What this does to the income statement
Traditional software lives at 80-90% gross margin because the cost of serving one more user approaches zero. A model-backed feature breaks that assumption: every use carries real compute cost. Publications through 2026 put the gross margin of such features in the 50-60% range.
The practical implication is not that the feature is unworthy, but that the pricing model has to move with it. Per-seat pricing, insensitive to usage, turns power users into a quiet deficit: when five percent of users generate most of the compute cost, a flat price simply subsidises them. We cover this separately in our article on pricing a model-backed product, and the link between per-unit cost and overall profitability in the unit model of a digital business.
How to measure it in practice
- Tag every API call with the identifier of the unit of work it serves, not only a user ID. Without that you cannot reconstruct cost per unit after the fact.
- Store input tokens, output tokens and attempt counts separately. A single cost total hides everything that could be fixed.
- Treat first-attempt success rate as a product metric, not an infrastructure one. It multiplies cost directly.
- Allocate fixed costs across units every month, or a growing feature will look worse than it is.
- Set a cost-per-unit ceiling as a release criterion, with the same standing as a quality criterion. A feature that breaches the ceiling does not ship.
That last point connects to a wider question: how do you define acceptance conditions for a system whose output is probabilistic at all. We set that out in our article on acceptance criteria for an AI feature, and the broader commercial side sits within our areas of expertise.
Frequently asked questions
What share of total cost of ownership is the API bill?
There is no single number, but in deployments with human oversight, an evaluation layer and monitoring, the API line is usually a minority of total cost. The more automated and less supervised the product, the larger its share becomes. That is why the seven lines have to be computed rather than inferred from a rate card.
How do you estimate cost before production exists?
Run a representative sample of several hundred real units of work through the full pipeline and measure tokens, calls and retries as they actually occur. Estimating from the prompt alone will nearly always understate, because it ignores retrieved context and failures.
Is a smaller model always cheaper?
No. A cheap model that fails more often produces retries, longer outputs and escalations to humans, all of which cost more than the token price gap. The correct comparison is cost per successfully completed unit of work, not price per million tokens.
When is it worth moving to a self-hosted model?
When the workload is narrow, repetitive and high-volume, and when API cost or latency has become the binding constraint. At low volume or on a broad task, the engineering and maintenance cost of running your own model usually exceeds the saving.
If you track only one metric, which should it be?
Cost per successfully completed unit of work, reviewed monthly. It absorbs token price, failure rate, escalations and retrieval efficiency, and it is the only figure directly comparable to your pricing and to the cost of the human alternative.