News

AI's Next Bottleneck Isn't the Model. It's Everything Underneath It.

• By John K. Waters
• 09/17/2026

For much of the generative AI boom, the industry's most visible competition has centered on models: which one is bigger, which one scores higher on benchmarks, and which one can reason its way through increasingly difficult problems.

At the AI Infra Summit in Santa Clara, Calif., this week, much of the conversation is about what happens underneath them.

More than 8,000 engineers, executives, architects, and other attendees have descended on the Santa Clara Convention Center for the three-day AI Infra Summit, which concludes Thursday. The conference has more than 400 speakers and more than 250 participating partners, spread across eight content stages and tracks covering compute, data movement, physical AI, data and models, and AI data centers.

The scale is itself revealing. Attendance at the conference, formerly known as the AI Hardware & Edge AI Summit, was about 3,500 last year, according to the organizers. This year's attendance has more than doubled as spending on the infrastructure needed to train and run AI systems continues to expand.

But the discussions in Santa Clara suggest the industry is entering a different phase of that buildout. The problem is no longer simply finding enough GPUs. AI developers increasingly have to worry about how efficiently those processors are being used, how quickly data can move between them, where the growing amount of model context is stored, and how much electricity the entire system consumes.

In other words, some of AI's hardest problems are becoming infrastructure problems.

From FLOPS to Tokens
Nvidia, unsurprisingly, has been one of the most prominent companies at the conference. But even its message illustrates the change.

Ian Buck, Nvidia's vice president of hyperscale and high-performance computing, used his keynote Tuesday to focus heavily on efficiency rather than raw processor performance.

Nvidia says the relevant measure for large AI systems is shifting toward the number of useful tokens that an AI factory can generate for a given amount of power. The company is now talking about "agentic tokens per megawatt," a metric that connects model output directly to one of the industry's most stubborn physical constraints.

The company says its DSX MaxLPS technology can deliver up to 1.4 times more tokens per megawatt through system-wide power optimization.

"Infrastructure that's fungible, that's reliable, that's going to last 10 years and really becomes an asset for computing the world's computing problems in the world's industries, they can build that with Vera Rubin, with DSX and all the innovations that we have here," Buck said.

Nvidia also announced that Amazon's Annapurna Labs is working with the company on NVHBM, Nvidia's custom high-bandwidth memory technology, while d-Matrix is integrating its inference accelerators with Nvidia's NVLink Fusion architecture.

The message is considerably broader than "buy more GPUs." Nvidia is increasingly presenting the AI data center as a single computing system in which processors, memory, networking, storage, software, cooling, and electricity all affect how much useful AI work gets done.

Inference Changes the Equation
Inference is accelerating that shift. Training frontier models still consumes enormous amounts of computing capacity, but once models are deployed to millions of users and AI agents begin performing longer-running tasks, inference creates a different set of infrastructure demands.

A model answering a single question is one thing. An agent repeatedly reasoning, calling tools, retrieving information, and maintaining context can generate a much longer-lived workload. That puts pressure on parts of the system that received less attention during the early race to build ever-larger training clusters.

Memory is one of them.

At the conference, SK hynix is showing several technologies intended to address the widening range of AI inference workloads, including processing-in-memory technology, High Bandwidth Flash, and a software system called SALT-KV.

"As AI workloads grow more diverse, ranging from ultra-low-latency agent services to long-context services where cost efficiency is critical, the conventional GPU-HBM architecture alone is increasingly unable to meet new requirements," said Lim Eui-cheol, vice president and head of Solution AT at SK hynix.

The problem becomes particularly acute with the KV cache, the stored information that transformer models use to retain context while generating responses. Longer conversations and agentic workloads can make those caches enormous.

SK hynix's SALT-KV approach is designed to manage that information across GPU memory, CPU memory, and SSD storage, deciding what information needs the fastest storage and what can be moved elsewhere.

High Bandwidth Flash, meanwhile, aims to combine high bandwidth with the much larger capacity of NAND flash for long-context inference workloads.

Those technologies remain emerging approaches rather than established replacements for today's GPU and HBM architecture. But their appearance at a major infrastructure conference illustrates how quickly the bottlenecks surrounding AI compute are multiplying.

The Network Becomes Part of the Computer
Moving data is becoming just as important as storing it. Modern AI clusters can contain thousands of accelerators that must exchange enormous quantities of information with very low latency. Adding processors does little good if they spend too much of their time waiting for data.

That is why networking and interconnect technologies occupy such a large part of the Santa Clara conference.

"The ability to move data efficiently between compute, memory and storage has become as important as compute itself," semiconductor company Marvell said ahead of the conference.

The summit also highlighted growing interest in silicon photonics and other optical technologies as conventional electrical connections hit bandwidth, distance, and power limits.

Nick Harris, CEO of optical-interconnect company Lightmatter, put the issue more directly during a conference session: "It turns out that the future of computing at this time, at the end of Moore's Law, at the end of Dennard scaling, it's all about networking," he said.

That represents another important change in how AI performance is measured. A faster accelerator does not necessarily produce a proportionately faster AI system if memory, networking, storage, or power cannot keep pace.

Then There Is Electricity
Ultimately, however, all of these systems encounter the same constraint: power. AI data centers are growing at precisely the moment utilities and communities are confronting how quickly they can build infrastructure to support them. That issue has become prominent enough that power and cooling are now core topics at an AI technology conference.

Alex Cordovil, research director at Dell'Oro Group, told attendees that expectations for new power capacity have changed dramatically. Where analysts once contemplated the United States adding roughly 10 gigawatts annually toward the end of the decade, he said projections now envision additions of 40, 50, or even 60 gigawatts per year.

That helps explain Nvidia's increasing emphasis on tokens per megawatt. A company can buy more accelerators. It cannot necessarily obtain another hundred megawatts of electricity on demand.

AI Becomes a Systems Problem
None of this means progress in AI models is slowing or becoming less important. Frontier developers continue to push model capabilities forward, and the compute required to train those models remains enormous. What Santa Clara shows is that the infrastructure around those models is becoming at least as consequential as what companies can actually deploy.

Intel, for example, is using the conference to promote heterogeneous computing and what it describes as disaggregated inference. AMD and SiFive demonstrated AMD's ROCm software running an AI workload on a system combining SiFive's RISC-V server processors with AMD GPUs.

Elsewhere on the show floor, vendors are attacking different pieces of the same problem: memory capacity, optical connections, storage throughput, networking, cooling, and power management. The result looks less like a race to build a single dominant processor and increasingly like an effort to redesign the computer around AI.

That may be the larger significance of the crowd in Santa Clara. A conference that drew about 3,500 people a year ago now attracts more than 8,000 to spend three days talking about the parts of artificial intelligence that most users will never see.

For the first few years of the generative AI boom, the model was the story. The next phase may depend just as much on everything underneath it.