Presented by F5
Model training is a batch problem, but agentic retrieval is a transaction problem. Unfortunately, most enterprises are now running transactional AI workloads on object storage they sized and operated for batch. That mismatch accounts for a significant number of production failures as AI agents and RAG (retrieval augmented generation) move out of pilot environments.
At the same time, enterprise AI budgets have concentrated on GPUs, while the storage tier underneath has received far less attention. Yet agents fundamentally change how that storage is accessed. AI agents and RAG pipelines issue continuous, high-concurrency calls for small objects instead of the large sequential reads a training job schedules in advance. That puts object storage directly in the path of every AI response: storage delays can increase the time users wait for an answer, while limits on simultaneous connections can prevent the system from staying responsive as agent activity scales.
"Training tells you what it needs before it starts, but agentic retrieval decides at runtime, so I don't know what they're going to ask for, when they're going to ask, or how much they're going to ask for," says Mark Menger, solutions architect for technology alliances at F5. "Nobody sizes a system for that, the way they sized for batch."
Agents invert the traditional access pattern
Training reads large objects sequentially from a known GPU cluster against a working set staged in advance, with one job on the system at a time. Agentic retrieval reverses nearly every one of those attributes. Traffic arrives as high volumes of small object GETs, mixed with PUTs, HEADs, and LIST calls that pull metadata. The client population carries high cardinality because agents spawn agents, and multiple tenants share the path by default. The client that decides what to request is a probabilistic system responding to input the enterprise does not control, which retires the application tier as a sufficient gatekeeper for what reaches the data.
Concurrency then compounds the latency each of those small requests carries. It can be likened to a busy bank lobby, Menger says.
"If nobody's in the lobby and one person goes up to one teller for one very large transaction, the system stays responsive," he says. "With agents, you have many people in line, a limited number of tellers, and those tellers only operate at a certain pace. Latency stops being a manifestation of how performant the system is and becomes a manifestation of how big that queue is."
Pilot traffic predicts nothing
Fan-out drives the volume: one prompt becomes several retrievals, those retrievals trigger tool calls, and the tool calls spawn more retrievals, so the ratio of requests to humans holds no stable value and pilot environments lose their predictive power. The potential number of agents and subagents they call, the number of tools all those agents call, the number of enterprise systems those tools invoke, and the number of requests they might make is staggering.
"Half a dozen agents in a lab environment run without incident," Menger says. "But unless you're very deliberately thinking about what scale looks like on day two, you're not going to see it in your pilot. Everything is fine until it's not, and when it's not, it's in a bad way."
Agents apply no backpressure
When storage is under stress, it returns a 429 and asks the client to try again later, which works when a human sits behind the request. Agents answer that signal by retrying automatically and in parallel, adding more load to the backend at the moment it has signaled that it has no capacity to spare.
"If it was just one client politely re-requesting every 10, 15, 20 seconds, that's one thing," Menger says. "But if you have 1,000 or 10,000 simultaneous clients all doing their retries at once, you're overwhelming the system at the exact time when it told you it needed space."
Organizations connect agents to the systems holding the enterprise data they need, so retry storms can hit infrastructure the business depends on. Across that shared infrastructure, a dozen clients issuing oversized or accelerated requests fill the queue while every other tenant sees timeouts, 429s, and 503s for traffic that never misbehaved.
What these failure modes look like in production
F5 saw the pattern at a global electronics manufacturer in Asia Pacific, where RAG applications and agents pull documents, images, and other inputs from storage clusters to support decisions on the manufacturing line. IoT devices simultaneously push telemetry into those clusters while AI consumers read heavily from them, and the AI data delivery layer has to determine which flow takes priority under load.
F5 reproduced both failure modes in its lab against a 32-node enterprise object storage cluster. With S3 clients wired directly to the cluster, the team induced misbehaving traffic and watched the cascade.
"It started with one or two nodes and then it rippled across," Menger says. "All of a sudden, the service just didn't respond anymore. It went from 'I'm struggling' to nobody responding at all."
Placing an application delivery controller, in this case F5's BIG-IP, between the clients and the cluster changed both scenarios. The controller intercepted the misbehaving traffic, well-behaved clients registered no measurable impact, and the cluster stayed healthy. When the team took two nodes offline, clients connected directly to the cluster saw error spikes and outright failures, while the controller steered traffic around the dead nodes. Traffic through the BIG-IP landed within plus or minus six percent of the direct-attached baseline, countering the standard objection that adding a control point between clients and storage slows everything down.
Protocol awareness makes the control point work
AI data delivery puts an intelligent traffic layer at the boundary where compute meets storage, and its usefulness rests on reading storage semantics rather than packets and ports. A controller that recognizes bucket, method, and tenant can route to particular clusters or nodes, enforce per-tenant limits and quotas in one place, and throttle by operation. LIST calls show why that granularity matters. Clients refreshing their view of a cluster all at once can cut cluster performance by roughly 75%, according to an F5 technology partner. But there's a line between that behavior and conventional load balancing.
"An application delivery controller doing bidirectional blast radius control pays close attention to the responsiveness and health of each node, proactively limits traffic to the unhealthy ones to give them breathing space, and sends traffic to the healthy ones, so the unhealthy nodes have a greater likelihood of recovery," Menger explains.
Fouad Chmainy, global solutions architect for AI and security at F5, says added capacity leaves the underlying architecture untouched." Capacity purchases leave those problems intact. More bandwidth does nothing for tail latency under bursty small-object load, and additional nodes behind one undifferentiated path enlarge the correlated failure domain."
Why more capacity can’t fix an architectural problem
Adding capacity also does nothing to distinguish between workloads. It cannot separate a legitimate retrieval from a runaway loop or prevent one tenant from consuming more than its share. Addressing those problems requires an architectural solution, Menger adds.
"Enterprises need a resilient and secure front door for storage," Chmainy explains. "Where a standard software-only load balancer in front of storage would previously do, more organizations are going to see the long-term value in the uptime, the resilience, and the security of this data if they architect for success."
That distinction becomes more important as enterprises move from experimenting with AI to operating it at scale: spending more on infrastructure is no substitute for an architecture designed to handle the resulting demands.
"Your ability to spend money does not necessarily equate to your ability to generate a well-engineered solution," Chmainy says. "Assuming I’m going to be really successful, what does successful look like at AI scale? It looks like nothing you’ve ever seen before. Connect this to all of your enterprise systems without the blast radius controls and you have the potential to really debilitate your business."
Sponsored articles are content produced by a company that is either paying for the post or has a business relationship with VentureBeat, and they’re always clearly marked. For more information, contact sales@venturebeat.com.