Presented by Microsoft
As AI portfolios expand, CIOs increasingly need to match each AI workload to the right combination of model, capacity, and infrastructure, while balancing utilization, performance, predictability, and cost.
Enterprise AI is rapidly becoming a portfolio problem. As AI portfolios expand, CIOs increasingly need to match each workload to the right combination of model, capacity, and infrastructure. A one-model, one-deployment approach becomes increasingly unrealistic when workloads differ in performance, utilization, predictability, and cost requirements.
Open models add more choice, but also more decisions around GPU capacity, infrastructure, and serving environments. The challenge for CIOs is therefore how to match workloads to the right combination of model, deployment approach, and infrastructure, without creating unnecessary complexity.
Different workloads need different deployment models
As AI portfolios expand, no single deployment model will suit every workload. Pay-per-token inference remains useful where flexible consumption matters, while dedicated capacity can make more sense for workloads with sustained demand or tighter requirements around capacity, latency, throughput, and cost predictability.
Pay-per-token | Dedicated capacity | |
Best suited to | Workloads where flexible consumption is important | Workloads where reliable capacity and more predictable latency and throughput are important |
Key consideration | Flexible consumption and minimal infrastructure involvement | Capacity, performance, and workload economics |
One way to address that trade-off is Managed Compute in Microsoft Foundry. As a managed deployment option, it allows customers to keep control over the decisions that shape inference performance and economics — including model, accelerator, capacity, and operating point — while Microsoft manages much of the underlying serving environment.
Customers can also use managed compute in Foundry alongside other deployment approaches as their workload requirements change.
Dedicated capacity changes the cost equation
Moving open-model inference into production changes both the economics and the technology choices.
Dedicated capacity means provisioning resources around anticipated workload requirements rather than relying solely on flexible, usage-based consumption. This gives teams a clearer basis for planning around available capacity, latency, and throughput. Budget owners also gain better visibility into infrastructure consumption and the relationship between capacity, utilization, performance, and cost.
As AI usage grows, that visibility gives organizations a firmer basis for comparing workloads by utilization, performance, predictability, and cost, not simply headline token or infrastructure pricing.
With dedicated capacity, the economics depend on how effectively provisioned GPU resources are used: GPU-hour cost, utilization, throughput, and the amount of work completed all contribute to the effective cost per token or inference request. That makes workload characteristics and expected demand central to the TCO calculation.
Less time on operations, more focus on development
Running open models on dedicated infrastructure can create significant operational work, from configuring and optimizing runtimes to maintaining software, addressing vulnerabilities, and applying security updates. A managed serving environment reduces much of that burden, allowing teams to retain control over the workload decisions that affect performance and economics, while spending less time managing the underlying infrastructure.
The right model needs the right infrastructure
Different models place different demands on infrastructure, and those differences can directly affect performance, GPU requirements, and workload economics, because GPU memory is consumed by both the model and the number of users it serves.
AMD Instinct MI300X provides a high-memory accelerator option within managed compute in Foundry, offering 192 GB of HBM3 high-bandwidth memory per GPU, or about 1.5 TB across an eight-GPU deployment. That capacity is what allows Z.ai GLM 5.3 Flash, a 320-billion-parameter model, to be served in Foundry on as few as four MI300X GPUs, with a 512K context window. And fewer GPUs for the same model means less infrastructure to provision and lower cost. This is all powered by ROCm, AMD’s open-source GPU software stack, which allows open models and the tools that serve them to run on Instinct in Foundry.
Accelerator choice is therefore critical to aligning infrastructure requirements, performance, and economics with the workload.
Portfolio decisions at production scale
AT&T provides a useful example of this in practice. Its OTel2.0 telecom-focused AI initiative used managed compute in Foundry to process massive volumes of industry-specific data and support different open models for different parts of the workflow.
“Foundry Managed Compute gave us access to GPU capacity at scale so our teams could focus on advancing OTel2.0 instead of managing infrastructure.” – Mark Austin, Vice President, Data Science and AI at AT&T
Rather than standardizing its AI workloads on a single model or infrastructure configuration, AT&T used multiple open models and GPU architectures across 530 GPUs. About 430 of those were AMD Instinct MI300X GPUs, processing roughly 1 trillion tokens through managed compute to generate synthetic training data. AT&T reported saving tens of millions of dollars by using open models for data generation rather than frontier-model consumption.
The significance isn’t just the scale of deployment, but the fact that different models and GPU architectures were used for different parts of the workload. It illustrates the portfolio-level decisions CIOs now face: how different combinations of model, infrastructure, and capacity can deliver the required performance at the most appropriate cost.
Managing AI infrastructure as a portfolio
As enterprise AI moves further into production, the strategic question is increasingly how to align each workload with the combination of model, deployment approach, and infrastructure that delivers the right balance of performance, predictability, utilization, and cost.
For workloads that benefit from dedicated open-model inference, managed compute in Foundry combines dedicated capacity with customer control over workload decisions in a managed serving environment. AMD Instinct MI300X adds a high-memory infrastructure option for demanding models and workloads, including optimized configurations for models such as Z.ai GLM 5.3 Flash.
For CIOs, that creates a clearer basis for managing infrastructure spend, improving cost and capacity predictability, and evaluating TCO and ROI as AI usage grows.
Sponsored articles are content produced by a company that is either paying for the post or has a business relationship with VentureBeat, and they’re always clearly marked. For more information, contact sales@venturebeat.com.