Ways to think about token pricing

There are only two things you can say with certainty about token prices: we’re in a supply crunch, and this is unstable. All of the variables are in play, and the market will get shaken out over the next few years to arrive at a new equilibrium. Right now we have a lot of frantic analysis of ‘time to power’, but the question at the end of that remains whether the foundation models have sustainable pricing power, strategic leverage and value capture, or whether they become low-margin commodity infrastructure providers. At the moment, I think every dynamic we can see points to the latter.

Clearly, the situation today is transitory. On the supply side, a trillion dollars or more of data centre capex is coming down the pipe (and plenty more semiconductor capex behind that), inference efficiency continues to improve very quickly, and new models are far more (or far less!) efficient in their token use. On the demand side, although the market has been capacity-constrained since 2022, the crunch in the first half of this year has been driven by sudden product-market fit in really just one use case, software development, and that’s actually a pretty small field (imagine if we had product-market fit for a consumer use case with hundreds of millions of DAUs - today’s infrastructure couldn’t support it at any price). We don’t know what the next use-cases to scale will be, nor when that would be, nor what their token needs would be.

Going up a level, it’s been pretty widely reported that inference today has 40-50% gross margins: this includes deprecation of the associated server costs (or the cost of renting them), but we don’t really know the asset life (five years? Seven years?) and obviously this doesn’t include the cost of training the next model a couple of times a year, which is currently far larger than revenue. In principle, inference is a marginal cost and training is a fixed cost, so with high enough revenue you can reach profitability, but we don’t know how training costs will change. On the other side of the table, it’s unclear how much of the surge in use in the last few months has an ROI (or at least has an ROI that can be quantified to a CFO), let alone any future use cases, and hence what prices people might be prepared to pay for them.

So, all the variables will move all over the place over the next 12 months, and move again over the next three to five years. How could we suggest where this will settle? How and where will supply, demand, price, capacity and capex get back into equilibrium?

In theory, you can model this bottom-up. You can make some assumptions around each of the variables I suggested above, and then try to model out how many chips there are now, how many chips with what performance TSMC and the rest of the semis industry might be able to deliver when, and how fast all of that can be brought online in data centers and how fast those can be powered. Then you can wonder about price discipline, and make some guesses about use cases. This will get you a number, but it will be rather like trying to build a five-year forecast for the broadband market in 1998: the spreadsheet will be very pretty, and you might even get close to the right number for this year, but there are too many unknown variables to make a useful forecast of a longer-term market structure.

In other words, we can say that token price is a function of supply and demand at a level between the sellers’ marginal cost and the buyers’ ROI, but we don’t actually know what supply, demand, marginal cost or ROI will be.

The other approach is to look from the top down: how do things like this tend to play out? What are the building blocks, and where can they go? Most of this conversation depends on what happens to this curve.

First, how many people will pay to be at the top right of the curve - to be at the frontier? At one extreme there are already use cases that already work just fine with a small, old, perhaps open source model that runs for ‘free’ on-prem or on your phone; at the other extreme there will be some that get better results from the latest, most expensive frontier model, consuming lots of tokens for lots of money; and then there will be many that are somewhere in between. So, how many use cases get better results from going how far up the cost curve, and how many have an ROI for that, and how much of the use is handled by models that are smaller, cheaper, good ‘enough’ and much more commoditised? The Panglossian view is that ROI might go up with more expensive frontier models because they have better results, but where does that really apply?

Second, does the frontier keep moving significantly? This is obviously the most basic science question in AI: how long does the frontier keep getting better, how long does that keep needing more and more compute, and does that continue to happen at a rate that keeps it ahead of downward pricing pressure from efficiency and capacity gains? Does the expensive head of the curve continue to be a thing?

Third, will there still be fierce competition between frontier models? Does the field shrink to fewer and fewer frontier models, perhaps with network effects emerging? Do frontier models diverge, with different models having much clearer leads in different fields? That could be another path to sustainable pricing power. Or do we continue with a mid-single-digit number of companies that are all making frontier models that all have generally equivalent capabilities? At the moment, everyone is using mostly the same science and mostly the same training data, and getting mostly the same results, and we don’t yet know of a network effect or any other winner-takes-all effect that would let one company pull ahead, stay ahead, and do things that others could not, in some sustainable way. Does that change?

Fourth, how much of the value from those high-end use cases is captured by the frontier model itself? How much needs to be wrapped in tooling, process, proprietary data, go-to-market, networks, support, and everything else associated with a traditional software company, even if you do need the big expensive frontier model underneath? Can that model do the whole thing, or is the model, no matter how good, still a piece of infrastructure that you use to make the actual product? At the extreme, can the model itself invent and make all of those things, and would that let them charge by seat, by outcome or just take the profit? Or do even (or especially) the most sophisticated and high-value use-cases need to sit inside hundreds of new companies that can pick and choose which models to use?

None of these are binaries: they're all a question of degree, and they'll probably vary quite a lot by use case. But at one extreme, there are two or three giant minds that run half of everything and have massive pricing power, and at the other extreme LLMs look like databases - there’ll be millions of them, some very big and some very small, and the value is in what you build on top - after all, every SaaS company is a ’database wrapper’. There’s a future in which Anthropic (or a company we haven’t heard of yet) wins the whole thing and can set its own terms, and a future in which dozens of routers run real-time auctions to allocate your tasks across hundreds of low-margin model-farms and a benchmark company takes a fee on every single one.

I don’t think anyone can actually know the answer yet. I’ve said “we don’t know” a lot, and that’s very deliberate. Part of the concept of the ‘S Curve’ is that there’s a stage early in the emergence of a new technology where it’s clear that this is going to be huge but nothing else is clear at all - the mid 1990s for the internet, say, and 2008 or 2009 for mobile. There are places where you can take a view - for example, I’ve argued at length that I think chatbots are a poor interface that will struggle to capture value up the stack - but we should presume there are big questions that we can’t see yet, let alone answer, and anyone picking one of the ten possible outcomes we can see and saying “it will be that one!” is just guessing.

Meanwhile, there is structural uncertainty at the early stages of every big new technology, but the uncertainty now is different, because we don’t have a good theoretical understanding of why these models work so well and so we don’t know how much better they can get. In 1995, we didn’t know how the internet would evolve but we knew that there were less than 100m PCs on earth (and they were expensive) and that telcos couldn’t give everyone FTTH next year; in 2010 we didn’t know what the next iPhone would be but we knew it wouldn’t have retinal projection. We knew the physical limits in ways we don’t really know with LLMs. Next month a new approach could cut inference compute needs by 90%, or double demand, or both.

All of this has people hunting for patterns to recognise (indeed, it’s been observed that all conversations about AI end in a hunt for metaphors). It has become common to draw comparisons with fiber, which had a massive overbuild in the Dotcom bubble that looks a bit like the infrastructure build-out today. The narrow problem with that is that fiber construction was far ahead of demand, where AI compute construction is far behind demand, though, as above, we don’t know what that supply/demand balance will look like in the future. But the more relevant objection, I think, is that fiber construction was mostly fixed cost (digging holes) rather than marginal cost (more equipment), whereas growth in compute needs means you need to buy more compute.

That makes mobile data a more fruitful comparison here. Mobile networks have marginal cost for capacity, and like AI they had an enormous surge in usage 15 years ago, that overwhelmed capacity and had carriers scrambling to add capacity and rebalance their pricing. Meanwhile, selling bits looks superficially similar to selling tokens: it's an opaque measure of marginal cost that doesn't map in any transparent or intuitive way to use cases or value, and needs to be replaced with bundles of some kind. But most importantly, in the last 20 years cellular data traffic has risen by several orders of magnitude, and this has become an enormous industry, with annual revenue of a trillion dollars and capex of $200 billion, but the stocks have gone nowhere, and all the value was captured by other people further up the stack. This, of course, is one of the core questions for AI: is this going to be low-margin commodity infrastructure with all the value captured by other people further up the stack?

Semiconductor manufacturing also has echoes of AI, since unlike cellular it has something of the same escalating cost and complexity that we see in foundation models. Rock's Law pointed out that the cost of a cutting edge fab doubles every four years: over time the frontier of semiconductors became so hard and so expensive that the number of players dropped from dozens to a handful and now really only one, TSMC. That’s another core question for AI: will this get so hard and so expensive that only a couple of people can do it, even without network effects? Equally, many semiconductor uses are further back along the same kind of price/performance curve I just discussed for AI. Even here, though, while TSMC has a de facto monopoly on the frontier, and nice margins, it doesn't actually capture a large share of value from the broader tech economy: net income last year was $53bn, less than half of Apple alone.

There are plenty of other comparisons one could make here: in the last six months Sam Altman has compared OpenAI both to Windows, a high-margin capital-light monopoly based on network effects, and electricity utilities, which are natural monopolies but also low-margin regulated utilities selling a pure commodity. One could also point to cloud (three leading players with good margins and clearly distinguished propositions, but again limited value-capture). But none of these have predictive value: analogies don’t have predictive value. You can't prove whether something will have the same outcome as mobile by arguing how much it’s like mobile: this was a mistake that many very clever people in tech made 15 years ago, claiming that Android would beat iOS because Android was ‘open’ and ‘open’ Wintel had beaten the ‘closed’ Mac in the 1990s (it’s also the mistake that doomers make when they claim that AI is ‘like’ nuclear weapons). Everything is different: bits, tokens and transistors are different, each of these examples are different from each other, and AI will be different too.

However, these examples do tell us, empirically, that something can be very important, very expensive, change the world, and be full of very sophisticated science and engineering, and yet have a wide range of possible outcomes. There isn’t one inevitable path here: you can have price equilibrium at high margins and at low margins, and with and without market concentration, and you can’t hand-wave that away by talking about AGI and saying “you don’t understand exponentials!”

However, if one thread in everything I've written above is how much we don’t yet know, the other thread is that every path to foundation models having market dominance, strategic leverage, value capture, winner-takes-all effects, or anything else other than becoming commodity infrastructure, requires something to change.

Maybe frontier models will become less competitive - yet in the last six months, Mark Zuckerberg and Elon Musk jumped from zero back onto the leaderboards. Maybe network effects will emerge. Maybe chatbots can grow into products and don’t need to be wrapped in software. Maybe one lab will start out-executing all the others and pull ahead on sheer product dynamism - Microsoft, Google, Facebook and Apple had to execute their way into a wining position before they had winner-takes-all efforts. Maybe something else will happen.

In particular, we have not one but two potential dei ex machina - Trump and China. China is reportedly considering regulating open source and some people close to Trump have floated this as well (though since Meta abandoned Llama the US has no leading open model), and export controls could expand and become systematic. Many people see the pleas for regulation from Anthropic and (sometimes) OpenAI as a front for regulatory capture, but either way, we can’t presume this will remain an entirely free market.

Even so, that brings me back to the same point: the current market dynamics point to a future in which, as today’s supply crunch eases, frontier models move towards becoming commodity infrastructure, with all of the value built on top, and for a different outcome, something needs to happen that we don’t see yet.