AI Bubble or Not

Getting specific about what bubble

Large language models that use transformers produce tokens that you then use to do things - generate images, interrogate Gemini for publications in a field, summarize something, generate simple code snippets - and this is only from personal experience. Demand for tokens is growing quickly according to public sources like Open Router. Whether there is a bubble in AI investment depends on whether people are willing to pay a price for those tokens that is higher than the cost to produce them and to cover the capital investment to produce them. So far, so very simple.

Many of my readers are familiar with the chain rule from calculus and even more may be familiar with the Kaya identity. To put some structure around discussions of AI bubbles and investment I like to use the following:

The first term is the cost of energy: how much it costs to generate electricity to run the chips and cooling in the datacenter.

The second term is how efficiently power is turned into floating point calculations and - more specifically for LLMs - the tensor operations and matrix multiplication.

The third term is a measure of how well software turns those floating-point operations into tokens.

This is a simplification but not an egregious one of the economics of inference or running the models. Whether there is a bubble comes down to how much people are willing to pay for tokens and whether that is more or less than that cost. Willingness to pay I will not cover here but there are some applications that are clearly valuable and for which there is a high willingness to pay: coding, investment research and the like come mind. There are other applications that are very token intensive that I am not sure there is much of a market for people to pay up for, not least of all those gorilla videos.

And a *checks notes* national socialist cartoon harassment campaign against a poaster.

What ultimately will make or break this business is getting those costs down.

On the power side: I have my doubts. In the US picking favorites in energy sources by blocking offshore wind and solar in an emerging power shortage seems unwise. I do not see orders of magnitude cost declines for any power sources from here. The outlook for help from the power and energy side is decidedly poor from a technologically point of view and made worse by the current policy mix.

On the efficiency side: I am much more optimistic. Photonics are likely to provide at least a few orders of magnitude in power efficiency gains here. Making this happen should be a core focus of the industry more broadly as the physical and political realities of required datacenter build become challenging.

On the software side: Improvements happen quickly and are continuing. The usual rule of thumb for models has been that inference FLOPs requires double the number of non-embedding parameters in the model per token. As model parameter size has exploded this has gone up a great deal. However, various improvements like mixture-of-experts, model distillation and smart routing have reduced this materially by not activating all parameters for every query. This is the equivalent of knowing that if you have a question about medieval literature that you want answered by a university faculty you should not send it to the physics department. This area remains one of active and fruitful development.

State-space models like Mamba will do even more as these approaches are integrated into services like Chatgpt. I expect gains to continue here and intensify.

So, is it a bubble? How will we know? How could it end? A few scenarios are below:

1. Token growth slows a lot: This is absolutely the worst-case scenario of demand collapse or materially falling short of projections. Everything gets hurt here, datacenters are empty or idled and AI is a maximal bust. I see this as wildly unlikely due to current token growth and increasing use. Not likely in my opinion but if token growth as measured starts to flatten or fall this case becomes likely. There is absolutely no sign of this happening at this time.

2. Productivity shock in model architecture or software: State space models have very compelling scaling properties that may significantly reduce compute demand and lead to reduced demand for datacenter and compute capital stock especially via reduced memory requirements. Brutal competition among models as the cost to compete goes down leading to an outcome like solar in the 2010s: the consumer wins, the producers do not. Good for downstream users of tokens but bad for whoever built all the capital. Ugly, but not out of the question especially if the willingness to pay for performance is somehow capped: Cursor seems to be dealing with the challenges of very expensive power users and very lucrative casual users. Debottlenecking this or making it economic to serve power users would be great especially if it led to internal model development being affordable and compelling. Impacts here are acutely demand sensitive as they may allow some applications to be served at a cost that is profitable.

3. Productivity shock in hardware: Very likely, and I have some knowledge but also some personal interests here. It should be noted there are a couple of plausible shocks here:

   1. Photonic compute: Vastly reduced energy use for compute, memory requirement impacts depend on subtle model choices. Great news for incumbent datacenters because you can produce more tokens in the same facility and with the same power fit out. It could be disastrous for the more speculative datacenter pipeline and create acute deflation in the space similar to the impacts of multiplexing on the value of fiber investments towards the end of the telecoms boom. This excellent paper Dynamic Analysis of the Long-Distance Telecom Bubble by Kurebayashi and Osgood gives a particularly thorough analysis of this boom and how the long lead capital expenditures on fiber that were expected to generate a certain throughput had vastly more capacity by the time of completion leading to oversupply.

   2. Interconnect and Memory Bandwidth: A gating factor for most LLMs today is how quickly data can be moved on and off memory or between clusters. This also drives lower GPU utilization as the GPU “wait” for data input / output. Have you ever found that you are consistently ready before your children, that you can mysteriously locate your shoes and socks in a timely manner? That is the life of a memory bandwidth constrained GPU. The problem with this debottlenecking is that as the GPU uses 90%+ of the energy in a rack then energy requirements explode without photonic compute. This is a gradual process and the development of coherent datacenter clusters will allow more compute per unit of datacenter albeit at an energy penalty. Nonetheless, this could also have elements of a supply shock in datacenters but would not necessarily imply that utilities are overbuilding right now.

There are plenty of bubbles that could be happening right now the further you get from the customer because the more links a business is away from the end token demand the more a productivity shock could upset the market. I am not particularly concerned about “hyperscalers” especially as they seem to be offloading some if not much of the credit risk onto alternative credit providers and specialist merchants like neo clouds. What I am particularly concerned about is a bubble in the infrastructure - something that is a pattern that has been repeated time and again in classics like Engines That Move Markets not just because of clear technological paradigm risk similar to the telecoms bubble but because the hostility and regulatory pressure is picking up and making their growth plans extremely challenging.