Running AI agents on your own hardware pays off financially only against expensive frontier models, and only above roughly 5 million tokens per day. Against open-weight model APIs, which sit around $0.14 to $0.50 per million tokens, the break-even is pushed out to billions of tokens per month and most organisations will never reach it. The implication is simple: if cost is the only consideration, the answer is usually no. The arguments that do hold are data sovereignty, version pinning and latency — and those need to be priced honestly.
The question most organisations ask is the wrong question
The discussion almost always opens with how much will we save by running this ourselves. That is a question you can calculate, which makes it feel safe — but it is almost never the real reason an organisation runs a model itself.
Three arguments hold water, and none of them is direct savings. The first is data sovereignty: when the information entering the prompt cannot leave the organisation, the economic question is secondary to the permission question. The second is version pinning: a provider who updates a model changes the behaviour of an agent calibrated to the previous version, and a model running on your own hardware does not change until you decide it does. The third is stable latency at high volume, particularly for classification and extraction tasks running hundreds of times per minute.
If none of those three applies to you, the arithmetic below will almost always send you back to the API.
The arithmetic, without magic
As of mid-2026, renting a single H100 ranges from roughly $2 per hour at dedicated GPU providers to around $14 per hour at the hyperscalers, with on-demand access averaging near $3 per hour. Buying a card costs $30,000 to $40,000 and up. Take a reserved price of $2 per hour — about $1,460 per month for a card running continuously.
| What you compare against | Price per million tokens | Monthly volume at break-even | Daily equivalent |
|---|---|---|---|
| Expensive frontier model | $10 | ~146 million | ~5 million |
| Cheaper frontier model | $2.50 | ~584 million | ~19 million |
| Open-weight model API | $0.25 | ~5.8 billion | ~195 million |
The table is computed directly: $1,460 per month divided by the price per million tokens. The third row is the one that ends the discussion in most organisations — 195 million tokens a day is the volume of a product at scale, not of an internal agent.
Note that this calculation is generous towards the hardware side. It assumes a reserved price, a single card, and zero operating cost. All three assumptions break in production.
Utilisation is the whole story
An API charges per token. A card charges per hour, including the hours when nobody sends it anything. That difference is why most self-hosting ROI calculations are wrong.
A quantised model in the 70-billion-parameter class, on a single H100 with efficient batching, reaches the order of 3,000 tokens per second. At $2 per hour, that translates as follows.
| Average utilisation | Effective cost per million tokens | Against an open-weight API |
|---|---|---|
| 100% | ~$0.19 | Competitive |
| 50% | ~$0.37 | Equal or more expensive |
| 25% | ~$0.74 | Three times more expensive |
| 10% | ~$1.85 | Seven times more expensive |
100% utilisation does not exist in the real world. Business load is cyclical — working hours, weekdays, month-end — so an average utilisation of 50 to 60 percent is considered very good for an internal workload. Below that, self-hosting loses both to open-weight APIs and to part of the frontier.
From this follows a simple rule of thumb: if you cannot point to a continuous workload that fills the card at least half the time, you are paying for air.
What the calculation leaves out
The price of the card is the only component that is easy to price, which is why it is the only one that gets priced. The full cost of self-hosting in production typically runs three to five times higher.
- Redundancy. One card is a single point of failure. An agent serving customers needs at least two, or a fallback route to an API.
- Operations. An inference server, monitoring, queue management, driver updates and re-quantisation on every model upgrade.
- On-call. Somebody has to answer when the GPU falls over at two in the morning. An API has an SLA; your card has you.
- Test infrastructure. Every model swap or quantisation change requires a full run of the test set, as detailed in the evals guide.
- Opportunity cost. A model you pinned a year ago stays where it is while the frontier moves on.
The architecture that does work: routing by task
The choice is not binary. A typical business agent performs a mix of tasks that differ sharply in character, and the economic logic differs for each.
| Task type | Characteristics | Where to run it |
|---|---|---|
| Classification, tagging, field extraction | High volume, short output, needs consistency | Small model on your own hardware |
| Summarisation and rewriting | Medium volume, medium quality suffices | Open-weight model API |
| Planning, reasoning, decisions | Low volume, high sensitivity to quality | Frontier model via API |
| Sensitive data that cannot leave | Regulatory constraint | Your own hardware, regardless of volume |
This routing delivers most of the saving without most of the risk: the high-volume tasks, which are also the simplest, drop down to a cheap local model, while the hard decisions stay with the best model available. Further techniques for reducing cost at the API layer are covered in the guide to reducing inference cost, and the last row of the table should be derived from your enterprise data classification.
Six questions before buying a card
- What is your daily token volume today, measured rather than estimated, and what is its growth rate.
- Which alternative price are you comparing against — frontier or open-weight. The two answers point in opposite directions.
- What utilisation do you expect across a day and across a week, including the hours with no traffic.
- Who on the team maintains the inference server, and what happens when they are on holiday.
- Is there a fallback route to an API when the hardware fails, and who decides to switch to it.
- Is there an argument that is not cost. If not, the decision almost always stays with the API.
This decision looks like infrastructure but it is architecture. It determines which parts of the agent can change and which are pinned, so it should be made against the full component map described in Anatomy of a Business AI Agent.
Frequently Asked Questions
Is it cheaper to run an AI model on our own server?
Only under specific conditions. Against a frontier model at around $10 per million tokens, the break-even for a reserved H100 sits at roughly 5 million tokens per day. Against an open-weight model API at around $0.25 per million, the break-even jumps to about 195 million tokens per day, a volume most organisations never reach.
What is the break-even between self-hosting and an API?
It follows from a simple division: the monthly hardware cost divided by the alternative price per million tokens. A card costing around $1,460 per month against an API at $10 per million breaks even at roughly 146 million tokens per month. Multiply the result by three to five to reflect full operating cost.
Why does GPU utilisation matter so much?
Because an API charges per token while a card charges per hour, including idle hours. Cyclical business load produces an average utilisation of 50 to 60 percent at best, so the effective cost per million tokens is almost always at least double the theoretical calculation.
What are the non-economic arguments for self-hosting?
Three main ones: data sovereignty when information cannot leave the organisation, model version pinning so a calibrated agent does not shift because a provider shipped an update, and stable latency on high-volume tasks. When one of these applies, the economic question is secondary.
Can you combine the two?
That is usually the right architecture. High-volume classification and extraction run on a small local model, summarisation on an open-weight API, and decisions and reasoning on a frontier model. This routing removes most of the cost without giving up quality where it decides the outcome.