GPUs running large language model inference can spend up to 60% of their compute recalculating context they already processed, says Alan McSeveney, field CTO at WEKA. Speaking with VentureBeat lead analyst Rob Strechay at the latest stop in the VentureBeat AI Impact Series, McSeveney said the waste goes back to how large language models hold conversation history in GPU memory, arguing that fixing it lets companies serve several times more users on hardware they already own. The efficiencies WEKA customers struggle with most show up in infrastructure, separate from the application metrics each business tunes for its own domain.
"The constraints people run into are budget, power availability and data center space," McSeveney said. "Those limits are getting in the way of a lot of organizations' attempts at growth right now, so everyone is trying to get more out of a capital investment they've already made."
Why GPUs recompute KV cache during multi-turn LLM inference
During multi-turn inference, GPUs store processed context in a KV cache so they don’t have to recalculate it with every exchange. But GPU memory is limited; when older context is evicted, the GPU may have to recompute the same history when a session resumes.
"Run without any strategy to deal with it, and a GPU can sit as low as 30% utilization on new work," he said. "Get a good strategy and you can see north of 90%, or 95% in the best case."
When a company builds its own infrastructure for AI workloads, GPUs take more than 80% of the budget, so a card that spends its cycles on recomputation is "a colossal waste of money," he added.
NVMe flash beats DRAM for KV cache offload
The industry is already moving toward more aggressive KV-cache reuse and offload, from Mooncake and recent DeepSeek techniques to Nvidia’s CMX context memory platform, built into its Vera Rubin stack. McSeveney argues that NVMe flash offers a more economical place to hold that cache than expanding server DRAM: It matches or beats DRAM for holding KV cache, scales much better and costs much less.
"You get that data off as quickly as possible, let somebody else occupy the GPU, and then get their data back on when they need it," he said.
The speed of that swap determines how many people one GPU can serve. If a system can move cache off and back on fast enough, each GPU can serve four to five times as many users. A company facing a capacity shortfall can fit that many more people or agents onto its existing servers before it orders another GPU.
How WEKA's Augmented Memory Grid shares KV cache across GPU servers
WEKA's Augmented Memory Grid, which runs on its NeuralMesh storage software, captures KV cache from GPU servers, stores it centrally, and loads it back into any host in the data center. A session can therefore stop on one server and resume on another without the second GPU recomputing its history.
Loading KV cache centrally raised the number of users on the same GPUs by 7x and lifted output from 200,000 tokens to 2 million in the same timeframe, compared with offloading to DRAM, McSeveney said. But optimizing the cache leaves the application side of the problem untouched.
"You still have to get your application, workflow, and pipeline as optimized as possible," he explained.
The question of whether to optimize model weights through quantization and fine tuning, or the harness around the model first varies by customer. But most teams with real LLM experience have already learned to curb hallucinations, cut waste and validate results. There's a larger knowledge gap in infrastructure, where many customers still do not know how much spare capacity their existing systems could free up for more queries and more users.
Capturing KV cache savings via GPUs or per-token APIs
It's not a question of whether to build a proprietary stack instead of a hosted service — the benefit goes to whoever controls the GPU, McSeveney said, which includes a company running its own hardware and a company renting raw GPU instances and operating its own software on them.
"If you're paying a foundation model provider per token, the majority of the benefit of any caching on their side goes in their pocket, not yours," he said.
That calculus becomes more important as GPU supply tightens. McSeveney pointed to Google's message to customers that every GPU it has ordered for Google Cloud but not yet installed already has a buyer for its full-service life, and that customers without a reservation by December should not count on on-demand capacity. Service providers buying from the same constrained supply will face the same squeeze.
"Running some of your own infrastructure might not be your first thought," McSeveney said. "But it may become a necessity."