Featured
Up to 60% of GPU compute goes to recomputing context. Flash storage can put that capacity back to work
GPUs running large language model inference can spend up to 60% of their compute recalculating context they already processed, says Alan McSeveney, field CTO at WEKA. Speaking with VentureBeat lead analyst Rob Strechay at the latest stop in the VentureBeat AI Impact Series, McSeveney said the waste goes back to how large language models hold conversation history in GPU memory, arguing that fixing it lets companies serve several times more users on hardware they already own. The efficiencies WEKA customers struggle with most show up in infrastructure, separate from the application metrics each business tunes for its own domain.
VB Staff
