How to run open models in production without owning the serving stack

Presented by Microsoft

Production inference requires technical teams to consider the model, deployment approach, and infrastructure together—without necessarily taking responsibility for the entire serving stack.

Running an open model in production means treating model choice, deployment approach, capacity, accelerator, and serving-stack ownership as a single engineering problem. Teams need to design around memory requirements, expected demand, latency, and throughput targets, and the operational boundary they are prepared to manage.

For technical teams, that means answering four questions.

1. Which deployment model best fits the workload?

Pay-per-token inference suits workloads that need flexible consumption without requiring dedicated infrastructure. Other workloads may benefit from dedicated capacity where sustained demand, predictable latency and throughput, GPU availability, or specific open-model and accelerator configurations are priorities.

This creates a middle ground for teams that need dedicated GPU capacity and greater control over model and infrastructure configuration but don't want to build and operate the full serving stack themselves. Managed Compute in Microsoft Foundry is one way to address it.

2. How should you design around capacity, latency, and throughput?

Once a workload runs on dedicated infrastructure, capacity and performance become engineering variables. Teams need to estimate demand, provision capacity, and choose an operating point that balances latency and throughput. Interactive applications may prioritize response time, while batch or high-volume workloads may optimize for throughput. When GPU capacity is allocated to the customer rather than shared, teams have a clearer basis for designing around those variables.

The economics change too. Instead of treating inference solely as a per-token consumption cost, teams need to consider the provisioned GPU capacity and how effectively they use it. GPU-hour cost, utilization, throughput, and work completed all contribute to the effective cost per token or inference request.

3. How much of the serving stack do you want to manage?

With managed compute in Foundry, customers retain the decisions that directly affect the workload, while Microsoft manages much of the serving environment.

This shift away from customer-managed infrastructure means that platform teams retain control over the variables that shape workload performance and economics, without having to maintain every layer underneath.

Because managed compute is part of Foundry, developers can also continue using familiar inference APIs, authentication, networking, SDKs, and observability across deployment types.

4. What accelerator characteristics does the workload need?

Memory capacity determines whether a model fits on a given configuration and how many GPUs it requires: additional memory means larger models can run on fewer GPUs, creating headroom for techniques such as batching, while memory bandwidth affects how efficiently data moves through the accelerator. Together with the serving configuration, these characteristics dictate concurrency, throughput, latency, and workload economics.

The AMD Instinct™ MI300X GPU is a good example: as a high-memory option in managed compute in Foundry, each GPU provides 192 GB of HBM3 high-bandwidth memory and up to 5.3 TB/s of memory bandwidth, with 1.5 TB across an eight-GPU platform. That capacity allows larger models to run on fewer GPUs, providing more headroom for batching and long-context inference, where model weights and the key-value (KV) cache compete for memory.

But hardware is only part of the equation. AMD’s ROCm ecosystem provides the open software and optimization layer around AMD Instinct, with wide support for open models and open-source AI frameworks. The scale of what this software alone can deliver shows up in AMD Performance Labs testing: on an 8× AMD Instinct MI355X, full-stack software optimization in ROCm 10 delivered an average 3.3× higher inference throughput across GLM-5, Kimi-K2.5, and DeepSeek-R1-0528, compared with a ROCm 7.0 baseline. In managed compute, the platform owns continuous optimization rather than offloading it to the customer.

GLM 5.3 Flash demonstrates how these characteristics translate into deployment choices. The 320B-parameter model, with 18B active per token, is available in three configurations optimized for AMD Instinct MI300X, each with a 512K context window. A four-GPU configuration targets compact chat, coding, and agent workloads, while two eight-GPU configurations use tensor (TP), data (DP), and expert parallelism (EP) to prioritize either concurrency and time to first token, or memory headroom for longer-context workloads.

The same principle can be seen at production scale. AT&T used managed compute in Foundry across multiple open models and GPU architectures, with about 530 GPUs, including around 430 AMD Instinct MI300X GPUs. Phi-4 alone processed more than 700 billion tokens a month for data preparation and synthetic data generation.

Configuration choices are important too. In tests of GLM 5.3 Flash using a conversational workload, an eight-GPU TP4 + DP2 + EP configuration delivered approximately 1.8× the output throughput of the four-GPU configuration, while more than halving average time to first token. Other workloads showed smaller gains, reinforcing why teams need to benchmark each configuration against their own workloads rather than base decisions on a headline GPU.

Accelerator benchmarks say relatively little about whether a configuration is right for a production application. Teams need workload-level measurements instead:

• How many GPUs are required for the model?

• What latency and throughput can be delivered at a defined operating point?

• How many concurrent users or requests can be supported while maintaining target performance?

• How effectively is the provisioned capacity being utilized?

• How does GPU-hour consumption translate into an effective cost per token or unit of work?

Production starts where model selection ends

Running an open model in production means matching the deployment model, capacity, management boundary, accelerator, and benchmarking approach to the workload.

For workloads that need dedicated infrastructure, the key decision is how much control teams need over model, accelerator, capacity, and performance without owning every layer of the serving stack. Managed compute in Foundry provides one way to strike that balance, with AMD Instinct MI300X GPUs providing the memory capacity to run large open models on fewer GPUs.

Sponsored articles are content produced by a company that is either paying for the post or has a business relationship with VentureBeat, and they’re always clearly marked. For more information, contact sales@venturebeat.com.