Presented by WEKA

Enterprises running multi-agent AI systems should measure them by cost-per-completed-outcome and keep each agent's working context in fast storage outside GPU memory, says Val Bercovici, chief AI officer at WEKA. Speaking with VentureBeat founder Matt Marshall at the VB Agentic Infrastructure Stack Summit, Bercovici pointed to two reasons enterprise GPU spending on AI agents runs higher than the work requires.

First is token-based billing. Model providers and their customers still price agent work by tokens consumed, so a team of agents looks more expensive than a single agent even when it finishes each task at a lower cost-per-outcome. Secondly, there’s recomputation of the KV cache, the short-term memory a large language model builds while it processes a prompt. When that cache falls out of GPU memory, the GPU serving the next agent in a swarm has to recompute context another agent already built, and enterprises buy extra GPUs largely to hold more of it in high-bandwidth memory.

"The number one rule is don't recompute the same token twice," Bercovici said. "Don't prefill twice."

Why enterprises should measure multi-agent AI by cost-per-outcome

Bigger agent teams aren't better. Independent AI evaluator Vals AI found that agent teams on its Vibe Code Bench AI waste huge amounts of tokens for a bump in speed, costing 1.8 to 5.1 times as much as single agents, with barely measurable quality gains. Customer conversations have shifted from "tokenmaxxing" at the start of the year to "token capping" now, as finance teams measure each agent run's cost against the value of the work it produced.

"We're still stuck as an industry on billing per token," Bercovici said. "We're billing consumption instead of billing outcomes."

Bercovici cited independent evaluators such as Vals AI and Epoch AI, which grade the quality of agent work in legal, financial, and medical tasks and calculate the cost of each completed task or session. For their numbers to hold up, these evaluators must stay independent of the model labs and agent harness vendors, and buyers should keep checking that they do.

"I think it'll be a higher volume of tokens at a lower unit cost, delivering that better cost-per-outcome," he said.

In a later panel, Bercovici and Upwork’s vice president of platform infrastructure engineering, Sudhanshu Bist, illustrated why that calculation is difficult. Upwork’s AI gateway tracks model spending and attributes it to teams and services, Bist said, but it does not capture training or fine-tuning costs, the human effort needed to correct results, or the CPU resources consumed by tool calls. Those costs sit in separate systems, making the full cost of an outcome harder to measure.

Why AI agent swarms need fast KV cache and long-term context retention

"There's never been such a thing as an agent in real-world production," Bercovici said. "It's always been plural."

Deployments either coordinate several agents or let one agent spawn subtasks that behave as agents themselves. As work passes between them, agents need access to the context already built.

Those handoffs draw on two kinds of memory. Short-term memory includes the KV cache, which stores the attention keys and values a model computes for each token. Keeping that cache lets the model generate subsequent tokens without reprocessing the entire sequence. Long-term memory preserves context from earlier sessions that an agent may need again when a user or another agent returns.

"The key word now is retention," Bercovici said. "How long are you going to need to retain information and access it very quickly, as opposed to recompute it?"

Agents that never stop make retention harder. "Black October," the recent wave of AI security breaches, exploited a 20-year-old PHP flaw, and researchers found the attack traffic seemed to come from automated agents running around the clock. In response, defenders now need their own agents running background tasks continuously, which means both memories stay in demand at every hour, with no idle window for a cluster to rebuild context.

Enterprises overbuy GPUs to get more high-bandwidth memory

Bercovici called GPU high-bandwidth memory and Cerebras’s on-chip SRAM “the most expensive real estate on the planet.” Scientific computing has always balanced floating point throughput against memory, and the two have fallen out of balance. Long contexts and agent swarms consume memory so fast that buyers end up paying for floating point capacity they don't use.

Capital seems to be the problem, Bercovici said, explaining that Chinese labs work under hardware restrictions, so they build leaner models and leaner inference stacks. U.S. companies, particularly in Silicon Valley, raise enough money to treat more hardware as the default answer. Nvidia CEO Jensen Huang's line, "The more you buy, the more you save," is misleading, in fact.

"Not really," Bercovici said. "The more you buy, the more you use."

DeepSeek's pricing has made that habit expensive. Competitors that match its per-token rates have to stop paying for duplicate prefill work, which means storing computed context somewhere cheaper than HBM and loading it back when an agent needs it.

AI inference moves KV cache from GPU memory to shared storage

Developers often blur two meanings of memory: when agent builders talk about memory, they usually mean a logical construct such as a database, a file system, a knowledge graph, or a vector store that feeds context into a prompt. The hardware meaning covers the state a model generates while it processes that prompt, starting with the KV cache.

That state moves through three hardware tiers. First is the GPU’s high-bandwidth memory: fast and expensive, with limited room for cached context as new requests arrive. The second is server memory, which provides more capacity but keeps the cached state on one machine. The third is shared flash storage, which can preserve context across sessions and make it available across the cluster.

Retaining context in shared storage lets compatible systems load previously computed state instead of rebuilding it, leaving the GPU to work on new tokens. Bercovici cited DeepSeek’s 100-pebibyte cluster to illustrate how much storage that working context can require.

"That is the working memory of AI, working long- and short-term memory of AI," Bercovici said. "It's not storage as you and I might have known it in the past."

Recovering GPU cycles for performance and efficiency

Software changes can also recover GPU cycles that current deployments waste. Mega kernels combine operations in the inference pipeline, reducing the overhead of passing work between them. Cerebras, SambaNova, d-Matrix and Groq have built dedicated inference chips that deliver thousands of tokens per second to a single user. Bercovici said mega kernels can help Nvidia and AMD GPUs approach those speeds when operators prioritize faster responses for individual users over serving more users at once.

"We're managing millions of cores in parallel per rack versus thousands in the past for CPUs," Bercovici said. "It's not that easy to orchestrate millions of things every few microseconds, so there's still so much performance and efficiency on the table that we're not addressing yet."