Ask why AI inference is so expensive and you will usually be handed a price list. The price list argues the opposite case: the published cost of a million tokens has fallen, repeatedly and on the record, for two years.
The bills still grow. Not the unit price — the total. That gap is the entire question, and it has a documented answer that requires nobody to be lying about their pricing.
The short version: the price of a token fell, and then three other things grew faster.
Key Takeaways
- The price cuts are published and real. Google cut Gemini 1.5 Flash input pricing 78%, to $0.075 per 1 million tokens, effective August 12, 2024.
- Nvidia says Grace Blackwell delivers “an order-of-magnitude lower cost per token” — in the same sentence calling computing demand exponential.
- Reasoning models changed what one question costs. OpenAI’s documentation states reasoning tokens are invisible to you and “are billed as output tokens.”
- A physical floor sits underneath. The IEA reports data centre electricity demand soared by 17% in 2025, against 3% growth in global demand.
The price per token really did fall
Google’s August 2024 announcement cut Gemini 1.5 Flash input pricing by 78% to $0.075 per 1 million tokens, and output pricing by 71% to $0.3 per 1 million tokens, for prompts under 128K tokens. Those are published list prices, not estimates.
The hardware layer tells the same story. In its fourth-quarter fiscal 2026 results, released February 25, 2026, Nvidia CEO Jensen Huang described Grace Blackwell with NVLink as “the king of inference today — delivering an order-of-magnitude lower cost per token.”
| Lever | Published figure | Source |
|---|---|---|
| Gemini 1.5 Flash input price | −78%, to $0.075 per 1M tokens | Google Developers Blog, Aug 2024 |
| Gemini 1.5 Flash output price | −71%, to $0.3 per 1M tokens | Google Developers Blog, Aug 2024 |
| Prompt cache reads (Claude) | 0.1× the base input token price | Anthropic documentation |
| Batch processing (OpenAI) | “50% cost discount compared to synchronous APIs” | OpenAI documentation |
| Grace Blackwell cost per token | “an order-of-magnitude lower” | Nvidia, Feb 25, 2026 |
The software side has moved too: OpenAI was reported to have halved its inference costs without new hardware — a claim never independently verified, which we flagged at the time.
So the deflation is genuine. That makes the growing bill a more interesting problem, not a lesser one.
What grew faster than the price fell
1. One question stopped costing one answer
This is the change most cost models missed, and it is documented in vendor manuals rather than in marketing.
OpenAI’s reasoning guide states plainly: “While reasoning tokens are not visible via the API, they still occupy space in the model’s context window and are billed as output tokens.” The same page says a request may generate “anywhere from a few hundred to tens of thousands of reasoning tokens,” and recommends reserving at least 25,000 tokens for reasoning and outputs.
Anthropic documents the same mechanism from the other side, directing developers to a thinking_tokens field that “reports how many of the billed output tokens were internal reasoning.” Its manual thinking budgets start at 1,024 tokens; for complex tasks it suggests 16,000 or more.
Read those two documents together and the arithmetic changes shape: the output you receive is no longer the output you pay for.
There is a second turn of the ratchet. Anthropic notes that on its newer models, prior turns’ thinking blocks stay in context and are billed as input. Reasoning you already paid to generate can be charged again, at input rates, on every later turn.
A 78% cut to the price of a token does not survive a task that consumes twenty times more tokens than it used to.
2. The number of questions is not fixed either
Unit economics assume a stable denominator. Agents removed one.
Huang’s February 2026 remarks are unusually direct: “Computing demand is growing exponentially — the agentic AI inflection point has arrived… Enterprise adoption of agents is skyrocketing.” Nvidia reported Data Center revenue of $62.3 billion for the quarter and $193.7 billion for fiscal 2026.
An agent does not ask one question. It asks, reads the result, and asks the next one — a loop in which cheaper tokens function as permission to run longer.
3. The floor under the price is physical
Software efficiency can compound. Electricity cannot, at least not on the same schedule.
The IEA reports that electricity demand from data centres soared by 17% in 2025, and that of AI-focused data centres climbed even faster — well outpacing growth in global electricity demand of 3%. It projects data centre consumption will double by 2030, with AI-focused power use poised to triple.
The same report notes capital expenditure at five large technology companies surged past $400 billion in 2025, set to rise a further 75% in 2026.
The filings agree: Microsoft’s quarterly capex hit $41 billion, two-thirds in short-lived assets (Microsoft AI Returns Face a Hardware Refresh Test), and Amazon guided to about $220 billion for 2026 (Amazon AI CapEx).
Complexity compounds further down the stack too, as we found in KLA AI Chip Inspection.
Two Lenses
Lens one: this is deflation working exactly as designed
The optimistic reading is that a falling unit price and a rising total bill are what a healthy technology transition looks like. Electricity got cheaper per unit for a century, and household bills still rose, because cheapness created uses that had not existed before.
Under this lens, reasoning tokens are not waste. They buy accuracy on tasks that shallower models simply failed. Paying more per query for a query that now works is a better deal, not a worse one.
The discount machinery supports this reading: cache reads at 0.1× base input and a documented 50% batch discount are real levers.
Lens two: the deflation is quoted, and the growth is paid
The cautious reading notes an asymmetry in what gets published. Price per token is announced. Tokens per task is not.
Neither OpenAI nor Anthropic publishes average reasoning-token consumption per request — precisely the figure that determines whether a price cut reaches you.
There is also a question of who collects. As we argued in Why AI’s Winners Aren’t the Companies Spending the Most, chipmakers are paid up front, while platforms carry the capital cost and wait for a return.
Why it matters
For teams budgeting AI, per-token pricing is nearly useless as a forecasting input. Cost per completed task is the only number that behaves.
For investors, it explains how demand and margin pressure appear together without either being fake. Falling prices expand the market and compress the take per unit at once.
What would change our view
We would revise this if vendors began publishing average reasoning-token consumption per request alongside list prices, and that figure proved stable rather than growing. That single disclosure would make per-token pricing forecastable again and undercut the core of the argument here.
We would also revise it if IEA-tracked data centre electricity growth fell back toward the 3% pace of overall electricity demand while AI adoption kept climbing. That would mean efficiency had finally outrun usage, and the physical floor had stopped binding.
FAQ
Q. Is AI inference actually getting cheaper or more expensive?
A. Both, measured differently. The published price per token has fallen — Google cut Gemini 1.5 Flash input pricing 78% in 2024, and Nvidia claims an order-of-magnitude lower cost per token with Grace Blackwell. Cost per completed task moved the other way for reasoning-heavy work.
Q. Why am I billed for tokens I never see?
A. Because reasoning models generate internal tokens before answering. OpenAI’s documentation states these “are billed as output tokens” despite not being visible via the API. Anthropic exposes the same figure through a thinking_tokens field.
Q. What actually reduces an inference bill?
A. The documented levers are structural. Anthropic prices cache reads at 0.1× base input tokens, and OpenAI’s Batch API carries a “50% cost discount compared to synchronous APIs” for workloads that tolerate delay.
Q. Is inference now bigger than training?
A. We could not verify a reliable public split. Figures circulating online trace to secondary sources rather than company disclosures, so we left the claim out.
Sources
- Gemini 1.5 Flash price update — Google Developers Blog
- NVIDIA Announces Financial Results for Fourth Quarter and Fiscal 2026 — NVIDIA Newsroom, Feb 25, 2026
- Reasoning models guide — OpenAI documentation
- Batch API guide — OpenAI documentation
- Extended thinking — Anthropic documentation
- Prompt caching — Anthropic documentation
- Data centre electricity use surged in 2025 — IEA
Leave a Reply