In VentureBeat Intelligence’s September VB Pulse survey, most companies running AI in production reported using half their GPU capacity or less — and most of those companies still plan to adopt new AI infrastructure.
Of the 102 qualified respondents, 68 were running AI in production and provided a GPU utilization figure. Fifty-five of those (81%) reported average utilization of 50% or less, and 71% of that group planned to adopt a new, additional or replacement AI infrastructure platform within 12 months.
Those findings raise a useful question about what buyers expect their next platform to solve. A replacement could improve scheduling, integration, or the economics of an existing workload. A new platform could support work the current environment handles poorly.
That question comes as Nvidia CEO Jensen Huang is urging companies to build their own AI factories. At his October 21 GTC Berlin keynote, he said, "To be part of this AI economy, you don’t want to just be consuming the AI; you want to be producing the AI."
The AI factory pitch is expanding beyond compute. HPE is emphasizing governed APIs and Model Context Protocol (MCP) integration, while VAST has announced an Agent Sandbox Foundation Stack with Nvidia designed to preserve agent state, data access and audit trails as the underlying compute scales up and down. In my conversation with Nebius ahead of the event, its security team highlighted GPU confidential computing as a way to protect data and model weights during processing and let customers verify the environment before releasing encryption keys.
My view is that integration and security are becoming more important in infrastructure decisions. Obtaining compute is one decision. Building an architecture that turns model capability into useful work requires several more.
As agents take on longer assignments, those decisions include how they access enterprise knowledge, preserve task progress, use different models, and recover when something goes wrong. Memory is an important part of that architecture. Cost per successfully completed task is one way to judge whether the pieces work together.
The evidence gap behind the next purchase
Across all 102 September qualified respondents, 63% planned a new, additional or replacement platform within a year. Only 10% reported an infrastructure ROI figure reviewed and signed off by finance. Among planned adopters, 91% had no such figure. Among respondents with no plans to change platforms, 89% also lacked a finance-reviewed ROI figure, a statistically insignificant difference.
Companies are making their next platform decisions before finance has validated the returns on their existing AI infrastructure.
Among respondents whose primary platform was in production, overall satisfaction averaged 4.0 out of 5.0, ease of implementation averaged 3.9, and value for money averaged 3.6.
A platform can work well enough to earn a favorable operational assessment while leaving a harder question unanswered: Is the work it enables worth its total cost?
For agentic applications, I would make cost per successfully completed task a central operating measure. That calculation should include inference, retrieval, storage, repeated attempts, and human review. Finance should then compare those costs with the value delivered against an agreed baseline.
A completed task needs a quality standard. A support case closed incorrectly should not count as a success, and the cost of repairing a code change should count toward the cost of completing it.
The architecture determines what the task consumes
September’s respondents identified several infrastructure-buying considerations. Integration with the existing cloud and data stack received selections from 48%, performance from 42%, and GPU availability from 35%. The difference between integration and performance was not statistically significant.
The architectural question is where a particular workflow loses time, repeats work, or consumes resources without improving the outcome. A more capable model may help. So might a better information path or context layer, a different serving configuration, or a clearer division of work between models and tools.
Memory is three connected design decisions
“Memory” can refer to different things, and treating them as one problem makes infrastructure decisions harder.
Institutional memory is the organization’s durable knowledge: policies, records, operating procedures, and past decisions. Agent task memory is the state needed to continue an assignment: what has happened, what remains, and what earlier actions produced. Inference memory refers to the resources and cached computation used during model serving.
Institutional knowledge must be current, relevant, and available under the task’s permissions. Task memory must preserve useful progress without allowing a temporary conclusion to become an authoritative enterprise fact by accident. Inference-memory techniques must be evaluated against the workload they are intended to improve.
Anthropic’s context-engineering guidance describes structured notes stored outside the context window as one way to preserve information during extended work. vLLM’s automatic prefix caching addresses a different layer: reusing cached computation when requests share a prompt prefix. Its documentation notes that this reduces prompt-processing work rather than the time spent generating new tokens.
September’s survey measures the inference-memory layer. Nvidia and Dell were the most frequently selected options for use or evaluation, at 41% and 39%, respectively, a statistically insignificant difference. Open-source KV-cache tooling and model-level efficiency techniques were each selected by 21% of respondents for use or evaluation. Another 16% had not addressed inference-memory limits or were unaware of them.
I expect the relationship between these forms of memory to become a more important architectural decision. Teams will need to determine what an agent retains, what it retrieves again, what it can share, and when a result becomes part of the organization’s durable record.
Where smaller models make economic sense
I have argued that small language models will form the long tail of enterprise AI. Their value will depend on where they run, which tasks they perform, and how they work with enterprise context, other models, and people.
The practical test is whether a smaller model, placed appropriately within the workflow, meets the required quality at a lower total cost.
For a bounded task, a smaller model may be sufficient. A more demanding step may require another model or human judgment. The system needs criteria for choosing that path and evidence that the resulting configuration meets the required quality and cost.
That evaluation belongs in the software development lifecycle. Teams should test model choice, information retrieval, task-state handling, and escalation together against representative work. They should repeat those evaluations when a component changes.
Token price is one input. September’s report found that 23% selected cost per million tokens as a buying factor, up from 11% in August, a statistically significant difference between the two monthly samples.
The final cost of a task also depends on token consumption, retries, tool use, and human intervention. Model selection should therefore be evaluated at the workflow level, with successful completion as the denominator.
Leaders and builders need a shared measure
The seniority results suggest that infrastructure discussions begin from different priorities.
Among vice presidents and directors, 52% selected total cost of ownership as a buying factor, compared with 4% of individual contributors. That difference is statistically significant under the report’s test for small independent groups. Individual contributors selected integration most frequently, at 58%.
Leaders and builders need to connect economic goals with the technical decisions that determine cost. Leaders need to understand the economic consequences of architecture. Builders need a cost metric tied to the technical decisions they can influence, so they can weigh performance against cost.
Cost per successfully completed task can provide that connection. It lets teams examine whether changes to retrieval, model selection, caching, or task-state management improve the outcome enough to justify the cost.
Find the bottleneck before buying more AI infrastructure
The next infrastructure decision should begin with a specific agent workflow and a clear understanding of what's limiting its performance.
Enterprise AI leaders should treat memory, integration and measurement as infrastructure design decisions. The place to start is:
Have the application or product owner define successful completion and acceptable human intervention.
Have the data team identify authoritative information and its access rules.
Treat observability as you would for any other application, with the platform engineering team measuring latency, utilization, caching behavior and total operating cost.
Involve finance, agree on the baseline and validate the resulting benefit.
Then change one part of the architecture at a time and measure the change in cost per successfully completed task.
Smarter models expand what agents can attempt. Smarter infrastructure, measured by the task, makes that work dependable and economically defensible.
We will take on inference economics, including how teams do more with less memory and fewer tokens, at VentureBeat's Agentic Infrastructure Stack Summit on October 28 in San Francisco. Applications to attend are open.
Research note: Survey figures come from the September VB Pulse AI Infrastructure and Compute report, published by VentureBeat Intelligence, with 102 qualified respondents at organizations with 100 or more employees, including consultants answering for their main client. Results are self-reported by a self-selected group. The utilization finding covers 68 production respondents who supplied a figure; 55 reported average GPU utilization of 50% or less. Companies reporting higher utilization number 13. The seniority comparison rests on 23 vice presidents and directors and 26 individual contributors.
The August and September token-price figures come from different groups of respondents; the difference between them is statistically significant. The inference-memory question combined use and evaluation. The institutional-memory, agent-task-memory, and small-model arguments are analyst interpretations rather than findings directly tested by the survey.