It's supposed to be easy. AI models keep leapfrogging one another in intelligence, so companies have set up gateways and routers that let their engineers switch models by changing a single line of code — allowing them to easily upgrade to whatever is the best model of the day.
Turns out, when companies try this inside an application that customers or employees already depend on, the swap can take weeks or months of engineering work. They often have to rework the application or go back to the previous model. A new best model arrives every few months, so companies that want to keep up face those weeks of work again and again.
VentureBeat asked about model swaps in its October survey of people who work on AI for companies with at least 100 employees. We asked whether the respondent's company had replaced the model in a production AI system in the past 12 months while keeping the surrounding application in place. Of all respondents, 47% said yes.
Of the respondents whose companies swapped a model, 70% said the swap took a few weeks or more of engineering work beyond changing the model itself. Of those same respondents, 36% said the swap took more than a month.
Only 13% of respondents whose companies swapped a model said the swap took almost no effort and was close to a configuration change.
The results of the swaps were mixed. Of respondents whose companies swapped a model, 55% said the new model performed as well as the old one or better. The other 45% gave one of two answers: performance dropped and their companies had to rework the surrounding application, or their companies went back to the original model. The 45% share has not changed in a statistically significant way since September.
To learn what makes a swap hard and how companies prepare for one, I drew on recent VentureBeat interviews and event sessions with leaders from Uber, PayPal, Home Depot, T-Mobile, Box and other companies.
Anthropic cut prices on Monday, and companies are routing work to cheaper models
Lower prices are one reason companies swap models, though our survey did not ask respondents why their companies swapped or whether the new model was cheaper. On Monday, Anthropic released Claude Fable 5.2 at $8 per million input tokens and $40 per million output tokens, 20% below the price of the version it replaces, as VentureBeat executive editor Carl Franzen reported. Our October survey closed before that release.
Companies can move work from mid-priced models to cheaper models through routers. A router is software that sends each request to a chosen model. Routing is one job of an AI gateway, which sits between a company's applications and the model providers.
Adobe's marketing agent uses a cheaper model to select the right context, and then Adobe sends the prompt to a larger model, said Anjul Bhambhri, senior vice president of engineering at Adobe, at VentureBeat's Agentic Infrastructure Stack Summit in San Francisco on October 28.
Some startups offer model routing solutions. Tetrate, one such company, has an "Auto Mode" selects the model for each request. Some Tetrate customers no longer care which model they get, said Varun Talwar, co-founder and chief technology officer of Tetrate, at the summit. Those customers tell Tetrate to "put whatever string you want" in the model setting, he said.
But a customer still has work to do before trusting that automatic choice, Talwar said. Tetrate is not yet "fully there" in knowing which model is right for which task, he said. Customers start with one team and make Auto Mode the default only if the answers are as good as before and costs have gone down, Talwar said.
So a gateway makes the model easy to change, but a company still has to check whether its application works as well with the new model.
Replacing the model inside an application that customers or employees already depend on is a different job, and it is the job our survey asked about. Sudhanshu Bist, vice president of platform infrastructure engineering at Upwork, an online marketplace for freelance work, drew the same line at the summit. For an open-source coding challenge, Bist said he personally would let a small model finish a task that a larger model started. In a production system, he would "probably stick to the existing model," he said. The choice of model depends on the task, Bist added in an email on Wednesday. Upwork sends repeated, well-defined work, such as matching freelancers to jobs, to smaller, tuned models and sends open-ended work to larger models, he said.
A production application can rely on prompts, tools and tests that engineers tailored to one model, and those parts may need rework when the model changes. The survey did not ask respondents which parts they reworked, or whether their companies use a gateway.
A new model can behave differently inside software built for the old one
Some companies have turned to fine-tuning models to customize them to work better in their domains. The company trains that model further on its own examples, but that fine-tuning does not automatically carry over to a replacement model. PayPal has had to change the model in an application "many times," said Ren Zhang, vice president and global head of AI product at PayPal, at a VentureBeat event on October 8. For PayPal's commerce agent, which helps shoppers research and buy products, PayPal moved to a smaller, faster model and fine-tuned the smaller model, which improved accuracy but not enough, Zhang said. PayPal did that work in internal research and testing, before the agent reached customers, the company said.
PayPal later decided to fine-tune less, because fine-tuning takes a specialist team and "when there's a new model, you have to now fine-tune again with the new model," Zhang said. PayPal now balances fine-tuning with out-of-the-box models and with how it allocates computing power, to get the best mix of accuracy, speed, cost and control, Zhang said through a PayPal spokesperson on Tuesday.
A new model can make an application worse because engineers tailor their software to the model they started with, said Ben Kus, chief technology officer of Box. Then the engineers change the model, and "a new model just maybe acts differently," Kus said on October 30 in an interview for VentureBeat's Beyond the Pilot podcast. Executives often assume that using a newer model means better results, and that is often not true, Kus said.
A company will probably see "the most severely different results based on the harness," he said. A harness is the code that wraps a model, feeds the model its instructions and controls which tools the model can call. The choice among the current frontier models makes less difference, because those models are all very good, he said. Box now tests harnesses and models in combination to find which pairing works, he said.
The harness can change what a task costs. Moonshot AI's Kimi K3 model passed 63.3% of the tasks inside both Claude Code and a DeepSeek Harness configuration, in Runta's FrontierHarness Eval benchmark. But the median cost per passed task was $18.34 inside Claude Code and $3.28 inside that DeepSeek Harness configuration, more than five times as much for the same score, the benchmark's leaderboard shows.
In FrontierHarness Eval, Kimi K3 ran through nine coding harnesses, in 12 setups, on the same 30 tasks. Runta, an agent infrastructure company that supplied the runtimes for the test, announced the results in early September. Thirty tasks is a small test, so the rankings are a rough guide, a point my podcast co-host Sam Witteveen makes in his video review of the benchmark.
Our advice to a company evaluating a new model is this: Test it in the harness it will actually use and, if the results disappoint, try another harness before ruling out the model. Otherwise the company may blame the model for a problem the harness caused.
For the largest models, the harness has "mattered less than we expected" in Artificial Analysis' research, said Micah Hill-Smith, co-founder and chief executive of the benchmarking company, at the October 28 summit. Smaller models, on the other hand, are less flexible, Hill-Smith said. A small model can show "20-, 30-, 40-point swings" in coding agent test results, depending on whether the model was trained on a particular harness's style, he said.
Moving to a smaller model is how companies make a task dramatically cheaper, Hill-Smith said, and the move has a price in engineering time. "We put in more work to allow us to use a smaller model," he said.
If small models react strongly to the harness, a harness that engineers built for the large model may not suit the small model.
A model that costs less per token can cost more per good result
A cheaper model can also cost more after the swap if it fails more often.
A small model that costs 10 cents per run and succeeds 20% of the time needs five runs on average, so it costs about 50 cents per good result, in a hypothetical example that Bist gave the summit audience. A larger model that costs 30 cents per run and succeeds 90% of the time costs about 33 cents per good result, Bist said in an email on Wednesday that clarified his figures. The larger model costs three times as much per run and still costs less per good result. The example assumes a company retries each failed run until one succeeds, Bist said.
Upwork's model gateway doesn't capture the full cost of a good result, Bist said. The gateway tracks spending on model calls by team. The gateway does not see the GPU computing used to train and fine-tune models, the time a person spends fixing an agent's result, or the processor cost of the agent's tool call, he said.
Some companies are starting to calculate cost per good result, rather than focusing only on cost per token, Tetrate's Talwar said. The AI industry is "still stuck" on billing per token when it should measure cost per outcome, said Val Bercovici, chief AI officer of WEKA, at the summit. WEKA was one of the summit's sponsors.
Some companies decide that a cheaper model is the wrong goal for a given job. T-Mobile uses the leading models for its customer service agents and pays a premium for them, said Jeff Simon, chief information officer of T-Mobile, at the summit. "We're not really trying to optimize for cost" in customer service, Simon said. T-Mobile is trying to give customers fast, accurate answers.
Uber says it can switch a managed agent's model in a couple of days
If an open-weight model offered the best return for Uber's AI code review, "we'll switch it in a couple of days," said Uday Kiran Medisetty, a distinguished engineer at Uber, at the summit. AI code review is one of Uber's managed agents, which are agents that Uber builds and runs centrally for one task.
Uber has a benchmark for code review, because Uber can test a model against any code change from past years, Medisetty said. Uber later built a benchmark for its background coding agent from real coding sessions, a job he said was much harder.
Uber uses what it learns from the benchmarks to pick the model for its automated code cleanup job, Medisetty said. That job removes stale feature flags. Uber cut its cost per cleanup by 90% by benchmarking and picking the right model, he said. Uber sees opportunities to cut costs in every managed agent, he said.
Uber also tracks the share of its AI requests that can move from one model to another, Medisetty said. Uber can move its managed agents from model to model; sessions that engineers run by hand in tools such as Claude Code are much harder to move, he said, because moving them means changing how people work.
What PayPal, Databricks customers, Box and T-Mobile and JPMorgan Chase do before a swap
PayPal keeps its business knowledge outside the model. PayPal's internal data science assistant relies on a knowledge layer that holds PayPal's business definitions, its approved queries and its step-by-step playbooks for analysis, Zhang said. According to PayPal, adding that layer raised the assistant's accuracy on internal business questions from 43% to 95% with the same model. PayPal built that layer separately from the model. "It's model agnostic because I purposely did it that way," Zhang said. When a better model comes out, "I'll just plug in and play. I should not change my knowledge." A company that ties its knowledge to one model will have to tune that knowledge again for every new model, she said.
Databricks customers turn a frontier model's answers into the test. Databricks customers follow a sequence, said Bryan Clark, director of product management at Databricks. The customers build the first version on a frontier model with a simple text index, which Clark called "smart agent, not smart back end." The customers save the frontier model's answers as an evaluation set. Then the customers move to a cheaper open-weight model, add vector search to help the cheaper model, and check the cheaper model's answers against the evaluation set, Clark said at the summit.
Box judges every change on three numbers: accuracy, cost and speed, measured on an evaluation set built from what customers ask Box's agents to do. "It doesn't matter how you did it, what will matter is those three numbers," Kus said. Collecting the evaluation data is usually the hard part, he said.
T-Mobile tests changes on real customer calls and watches every session afterward. When T-Mobile's engineers want to change how its customer service agents are set up, T-Mobile runs the old and new versions at the same time and compares them on real customer calls, Simon said. T-Mobile scales the new version only if it works better. T-Mobile also runs a small model that monitors every chat and voice session and flags answers that look wrong, he said. Testing before launch cannot cover every way a customer asks a question, so T-Mobile looks for the unusual cases "that you'd never see in a lab," Simon said.
JPMorgan Chase approves new models before employees get access. Another thing that slows down model switching is organizational governance. JPMorgan Chase, which is rolling out Anthropic's Claude to more employees, makes every new Claude model pass the bank's own approval process before anyone at JPMorgan Chase can use it, Anthropic said on Monday.
What companies can do before their next model swap
The work a swap takes is a reason to evaluate a switch carefully, not to rule it out altogether. Beliefs about AI that held three or four months ago are very likely no longer true, said William Allen, Snowflake's head of product for CoCo, its AI coding agent, and CoWork, its AI assistant for knowledge workers, in a conversation with VentureBeat on November 3. "Probably the most important fallacy for any and all of us to understand and really grok is the sunk cost fallacy, where you said, no, no, no, I spent all this time building this thing, I don't want to rebuild it," Allen said.
Based on what these speakers described, the takeaway is: Do the preparation before the swap.
First, build an evaluation set from real requests and answers checked for accuracy, so the company can test a new model against it.
Second, compare models on cost per good result, counting the failed attempts. Test each model inside the harness it will run in and, if possible, in at least one other harness.
Third, keep business knowledge, such as definitions and rules, in a layer the company controls, so engineers do not rebuild it for each model.
Fourth, compare the old and new versions of the application on real traffic, and keep monitoring after the swap, because tests do not cover everything users ask.
VentureBeat will ask the same questions about model swaps in its November survey and report whether 70% of respondents whose companies swapped a model still say the swap took a few weeks or more.
About the data: VentureBeat Intelligence ran this VB Pulse survey on agent orchestration in October 2026. Respondents answered about an organization with at least 100 employees, and 28% of respondents are consultants who answered about the client where they spent the most time. Only respondents who reported a model swap answered the question on engineering effort, and the percentages on swap results describe the same group, so percentages for that group are less precise.
For this comparison, September's figures include only respondents who would have passed October's eligibility screening. In September, 49% of respondents whose companies swapped a model said the swap took a few weeks or more, and 12% said the swap took more than a month. VentureBeat Intelligence labels both October increases as directional, because the mix of respondents changed. In October, 35% of respondents said they use AI systems or track them from a business perspective, a much larger share than in September, and those respondents chose "more than a month" more often than other respondents. Without those respondents, 66% of October respondents whose companies swapped a model said the swap took a few weeks or more, and 29% said the swap took more than a month. Most October respondents came through a research panel, and we cannot tell how much of any change from September reflects who answered. You can find more VB Pulse reports here.