AI Compute Costs: The Great Divergence

by

Key Takeaways

AI compute costs are splitting in two opposite directions. The cost to ‘build’ a frontier model has risen more than twofold every year since 2016, putting a single training run on track to exceed $1 billion by 2027 if the trend holds, while the cost to ‘use’ a fixed level of capability has collapsed roughly 10x per year — a 1,000x drop in three years. Drawing on primary data from Epoch AI, arXiv, and a16z, we unpack what this divergence means for the structure of the AI industry.

  • Training compute has grown 5x per year since 2020, doubling about every 5.2 months (Epoch AI)
  • Amortized training cost has risen 2.4x per year since 2016 (90% CI 2.0–2.9x), extrapolating to $1B by 2027 (arXiv, Epoch)
  • Fixed-performance inference prices fell 9x–900x per year; a16z estimates a 1,000x drop in three years (Epoch, a16z)
  • Our read: value flows to the poles — frontier concentration and inference democratization (a barbell)

A billion dollars to train one model. Yet the price of running that same intelligence is sliding toward almost nothing. Both numbers come from the same industry’s measured data.

The debate over AI compute costs usually picks a side — AI is too expensive, or AI is suddenly too cheap. Lay out the primary data, though, and both claims turn out to be true at once. The cost of making a frontier model and the cost of running one are tracing exponential curves in opposite directions. That is the divergence this analysis takes apart.

AI-generated illustration: an amber curve rising toward a dark server tower and a blue curve descending across a data-center floor
AI compute costs split into a rising ‘build’ curve and a falling ‘use’ curve — AI-generated illustration (OpenAI gpt-image-1-mini)

AI Compute Costs: Two Exponential Curves Running Opposite

Training compute for frontier language models has grown 5x per year since 2020, doubling roughly every 5.2 months, according to Epoch AI. That curve only points up. More compute means a bigger bill to buy it.

The opposite curve is inference. The price of running a fixed level of performance keeps falling — Epoch reports declines ranging from 9x to 900x per year across tasks. One line climbs, the other drops, and that is the exact shape of the divergence.

Here is why this is not a simple cost-cutting story: the two curves measure different things. One tracks conquering a performance frontier no one has reached before, while the other tracks serving a capability already in hand.

Key point: The same phrase, ‘AI compute costs,’ holds two economies that move in opposite directions — one up, one down.

Building It: 2.4x a Year, Toward a Billion-Dollar Run

Epoch AI researchers (Cottier et al., arXiv:2405.21015) estimated training costs for 45 frontier models using three methods. The amortized hardware-and-energy cost has grown 2.4x per year since 2016, within a 90% confidence interval of 2.0x to 2.9x. A cloud-rental calculation lands at 2.6x, and both point steeply upward.

Extend that line and the implication is blunt. The researchers wrote that if the trend continues, the largest training runs will cost more than a billion dollars by 2027. This is a direct extrapolation that assumes the historical growth rate holds, not a settled forecast.

a close up of a server in a server room
More than half of a training bill comes from accelerator chips and servers — Photo by Tyler on Unsplash

The cost mix is revealing. Across GPT-3, OPT-175B, GPT-4, and Gemini Ultra, accelerator chips, servers, and interconnect made up 47% to 67% of the bill, while R&D staff — equity included — took 29% to 49%. Energy itself was only 2% to 6%. In other words, the engine of rising training costs is silicon and talent, not the electricity meter.

The takeaway for anyone watching the industry is clear. Building the frontier is becoming a capital-and-chips game, and the entry bar more than doubles every year.

Using It: 1,000x Cheaper in Three Years

Now flip the curve. Measured at a fixed performance level, inference prices have fallen between 9x and 900x per year, depending on the task. Epoch puts the median at 50x per year, and at 200x per year when you look only at data since January 2024.

A concrete case helps. On PhD-level science questions (GPQA Diamond), the price to reach GPT-4’s accuracy dropped 40x per year. The money needed to buy the same answer shrank to one-fortieth annually.

Venture firm a16z frames it even more plainly. For equivalent performance, inference cost has fallen about 10x per year — a 1,000x drop between November 2021 and November 2024. The price of a million tokens at GPT-3 quality fell from $60 to $0.06 on Llama 3.2 3B.

In simpler terms: The answer that cost supercomputer money last year costs pocket change on a commodity chip this year. The price tag on a fixed level of performance shrinks by a single-digit fraction annually.

This holds even as reasoning models spend more tokens ‘thinking’ before they answer. When the per-token price keeps collapsing, even token-hungry techniques grow affordable over time.

Not a Contradiction: ‘Frontier’ vs ‘Fixed Performance’

So how can the cost of making rise while the cost of using falls? Put simply, the curves measure different targets. Training cost chases a frontier that gets harder each year, while inference cost serves a capability already fixed in place.

Two forces drive the fixed capability cheaper. Pre-training compute efficiency improves 3.0x per year, so the same performance needs one-third the compute annually, per Epoch. Hardware price-performance, measured in FLOP/s per dollar, has added roughly 37% a year from 2012 to 2025.

Our read — In our analysis, this divergence is not a paradox but the natural result of a clock that turns yesterday’s frontier into today’s commodity. Tie three separately reported figures into one causal chain and it reads like this.

  1. Because training compute grows 5x per year, the cost of building the newest frontier rises 2.4x per year (Epoch, arXiv).
  2. Because algorithmic efficiency (3.0x/year) and hardware value (1.37x/year) compound, the cost of using a fixed performance level falls fast (Epoch).
  3. So at any moment top performance gets pricier while one-to-two-year-old performance collapses — a16z’s observed 10x annual drop is the result.

Why this matters: only by separating the two curves does the ‘AI is expensive vs. cheap’ argument resolve. For anyone chasing the frontier, AI gets dearer every year, and for anyone content with last year’s capability, it gets cheaper.

blue and black circuit board
Algorithmic efficiency and hardware value are the two forces dragging inference prices down — Photo by Sahand Babali on Unsplash

The Divergence at a Glance: A Doubling-Time Table

Pull the headline figures into one place and the divergence sharpens. The table below organizes the primary numbers from Epoch AI, arXiv, and a16z by direction and rate of change. It is this article’s synthesis of values the sources never placed side by side.

MetricDirectionRate of changePrimary source
Frontier training computeUp ↑5x/year (2x every 5.2 mo)Epoch Trends
Training cost (amortized)Up ↑2.4x/year (since 2016, 90% CI 2.0–2.9)arXiv, Epoch
Training power drawUp ↑~2x/yearEpoch Trends
Inference price (fixed performance)Down ↓9x–900x/year, median 50xEpoch
Inference price (equivalent performance)Down ↓~10x/year, 1,000x in 3 yearsa16z
Pre-training compute efficiencyImproving3.0x/year (2x every 7.6 mo)Epoch Trends
Hardware price-performance (FLOP/$)Improving1.37x/year (2x every 2.2 yr)Epoch Trends

The top three rows and the bottom two point in exactly opposite directions. Bridging them are the two ‘improvement’ metrics at the bottom, the engine that drags fixed-performance prices down.

The Barbell Market — Who Wins

Translate the data into market structure and a barbell appears. At one end sit the few frontier labs and nations that can absorb training costs rising 2.4x a year toward billion-dollar runs. The cost bar itself becomes the moat, concentrating top performance among a handful of players.

At the other end sits the application layer, absorbing collapsing inference prices as margin and reach. You do not have to train a model to pull last year’s capability almost for free.

Our read — In our analysis, the most dangerous seat is the middle of the barbell. A business whose moat is a mid-tier model itself is fully exposed to the commoditization clock.

  1. Inference prices fall 10x a year, so this year’s ‘average model’ is next year’s cheap commodity (a16z).
  2. Meanwhile frontier entry costs rise 2.4x a year, so a middle player cannot easily vault to the top (arXiv, Epoch).
  3. An undifferentiated mid-tier model is squeezed from both ends — value flows to the poles.
Disclaimer: This article is for informational purposes only and is not investment advice. It does not recommend buying or selling any specific security or digital asset. All investment decisions and their outcomes are the sole responsibility of the reader.

2027, If the Curves Hold

Assume the curves hold and training’s real constraint shifts from algorithms to capital, power, and chips. Epoch reports that the power draw of frontier training roughly doubles each year, with the latest runs already pulling tens to hundreds of megawatts. The largest known run is Grok 4 at about 5×10^26 FLOP.

Inference runs the other way. Once cost effectively drops out as a bottleneck, the competitive axis shifts from running price to differentiation. Everyone can run the same capability for almost nothing. What remains is data, product, and distribution.

None of this is destiny — it is the extension of a past curve. Saturation of scaling gains, a data ceiling, regulation, or a sudden jump in efficiency could each bend the slopes. Read the extrapolation for what it is: a conditional, not a promise.

That narrows the watchlist to two numbers. First, whether the disclosed cost of the largest single training run actually reaches the billion-dollar mark; second, whether the annual decline multiple for fixed-performance inference holds near 50x. Those two figures will write the next chapter of the divergence.

💡 Related Posts
AI Semiconductors Own the Rack — the $1.2T Teardown
💡 Related Posts
Reasoning Models Explained: How AI Thinks

References

Frequently Asked Questions

Q1. Why do AI training and inference costs move in opposite directions?

Training cost chases an ever-harder ‘frontier,’ so compute and capital inputs more than double every year and the bill rises. Inference cost serves an already-fixed level of performance, so algorithmic efficiency (3x a year) and hardware value (1.37x a year) compound to make it cheaper fast. The targets differ — a moving frontier versus a fixed capability — so it is not a contradiction.

Q2. Will training an AI model really cost $1 billion by 2027?

Epoch AI estimates that extending the 2.4x-per-year growth in amortized training cost since 2016 (90% CI 2.0–2.9x) puts the largest single run above $1 billion by 2027. That is an extrapolation built on the assumption that the trend holds, and saturation or an efficiency jump could change it.

Q3. Is the 1,000x drop in inference cost real?

Per a16z, equivalent-performance inference cost has fallen about 10x per year, a 1,000x decline from November 2021 to November 2024. One example is the price of a million tokens at GPT-3 quality falling from $60 to $0.06 on Llama 3.2 3B. Epoch reports task-level declines of 9x to 900x per year, with a median near 50x.

Q4. So who actually makes money in AI?

Our read is that value concentrates at the two poles of a barbell. One end is the handful of frontier labs that can fund billion-dollar training runs; the other is the application layer that absorbs collapsing inference prices as margin and reach. The riskiest seat is a business whose moat is a mid-tier model itself.

Q5. Is power the real bottleneck for AI scaling?

Epoch reports that the power draw of frontier training roughly doubles each year, with the latest runs pulling tens to hundreds of megawatts. Energy is only 2% to 6% of the training bill, but if the curve holds, securing power could join chips and capital as a real constraint on the training side.

Stay Ahead of the Curve

Get the latest insights on AI, tech, and markets — delivered in both Korean and English.

Explore More Posts