The harness surrounding an AI coding model — its system prompt, tools and context-management rules — can triple the cost of solving coding problems without producing a measurable improvement in accuracy, according to new research. AI vendors regularly advertise accuracy gains from harness updates. But whether a more elaborate harness actually makes an agent better at solving coding problems is less clear.
Researchers at Shandong University put that question to the test. They compared three production harnesses by holding the underlying models fixed and testing them on nearly 500 software engineering tasks. On the larger task set, the most elaborate and most minimalist harnesses produced statistically equivalent success rates within a five-percentage-point margin, although a third harness performed worse.
On the 45 hardest tasks, switching harnesses changed which problems the agents solved about as often as rerunning an unchanged setup: 13% of tasks in either case, measured at the median. Meanwhile, the researchers found that cost per task varied by as much as threefold across harnesses using the same model.
For enterprise technical leaders and developers, the findings are a warning against choosing coding agents based on small, single-run benchmark differences. Teams need to look at both task success and cost per task.
Why harness benchmarks can mislead
Public leaderboards often compare coding agents running different harnesses, making it difficult to isolate what the harness contributes. Some studies report that keeping a model fixed and swapping harnesses can result in spreads of 23.8 points on a graded task score and 27.4 percentage points in Pass@1. Meanwhile, Zhipu AI advertises a 2.4-point gain for its ZCode harness over Claude Code, using the same model.
The new study warns that these numbers are difficult to read for two reasons. First, many published comparisons run each configuration only once, while rerunning the same tests under the same conditions can yield widely varying results. A separate study cited in the paper found that 10 reruns of the exact same setup on SWE-bench Verified resulted in score spreads of 2.2 to 6.0 percentage points. Without rerunning the baseline configuration, teams cannot tell whether a score difference comes from the harness or from random run-to-run variation.
Second, the harnesses being compared are often not on equal footing. In one study cited in the paper, the same harness scored 19.1% with a minimal benchmark adapter and 73.4% with the full adapter, using the same underlying model.
Test conditions can distort results, too. When the researchers audited early web-enabled runs, they discovered that 312 of 439 DeepSeek and Qwen trials (71%) had retrieved the original human-written fix from GitHub. The researchers discarded those runs and blocked network access in their final evaluation.
Testing production harnesses head-to-head
To measure what a harness contributes, the researchers tested three production harnesses: Claude Code, OpenCode, and mini-SWE-agent, a minimalist open-source harness written in about 100 lines of Python that exposes a single bash tool and performs no context compaction.
The researchers ran five models through all three harnesses: two locally served open-weight models (Qwen3.6-35B-A3B and Qwen3.8-27B) and three API-served models (DeepSeek-V4-Flash, GLM-5.3-Flash, and HY4-Preview). They also used Claude Opus 5 inside Claude Code as a top-tier anchor.
They evaluated the agents in isolated containers across 492 offline tasks from SWE-bench Verified: the 45 hardest tasks, which take humans one to four-plus hours, and the remaining 447 tasks. All five models were tested on the 45-task subset, but only the two Qwen models were tested on the larger pool.
The more elaborate harnesses showed no consistent accuracy advantage. On the 447-task pool, heavyweight Claude Code and minimalist mini-SWE-agent scored within two percentage points of each other and were found to be statistically equivalent within a five-point margin on both Qwen models. On the 45 hardest tasks, roughly one out of every eight problems (13%) switched between passing and failing when switching harnesses. Rerunning the same harness produced the same median switch rate. Even testing DeepSeek’s official command-line harness on its own DeepSeek-V4-Flash model produced a pass rate within the range of the other three harnesses — and the highest cost per trial of any harness tested on that model.
“A vendor’s own harness on its own model buys no score advantage,” the researchers write.
The only harness effect that cleared the statistical noise floor was a loss. OpenCode trailed by up to 9 points on the large task pool. On Qwen3.8-27B, roughly half that gap was associated with runs cut short when the model hit the 32,000-token output limit mid-step, often before it could write a code patch. Claude Code hit the same cap on the same model, but automatically prompted the model to continue.
While the more elaborate harnesses showed no consistent accuracy gains, harness choice had a substantial effect on inference costs.
“With the same model, the same tasks, and one price list, cost per task differs by up to 3x across harnesses,” the researchers write.
On GLM-5.3-Flash, a trial in Claude Code cost nearly twice as much as in mini-SWE-agent and over three times as much as in OpenCode. However, OpenCode also stopped after fewer model calls, contributing to its lower cost. All three harnesses returned roughly the same amount of text from tool calls. The difference also began with the first API call, where the harnesses sent very different amounts of fixed context through their system prompts and tool schemas: 16,581 tokens in Claude Code, 7,025 in OpenCode, and 829 in mini-SWE-agent. Because agents re-send the conversation transcript on every turn, that initial overhead is multiplied by the number of steps, and the researchers infer that Claude Code adds further weight by resending the model's intermediate reasoning blocks.
“What separates the harnesses’ per-step weight is their own text, not the repository’s,” the researchers explain.
Even though 96% to 99% of input tokens in these loops hit the provider's prompt cache, repricing the tokens across different vendor cache rates scaled the bill without changing which harness was most expensive for a given model.
The cost ranking also changes with the model. On DeepSeek-V4-Flash, mini-SWE-agent took 141 model calls per trial to Claude Code's 104 and cost slightly more — about 15.2 cents per trial versus 14.6 cents.
Confirming the harness tax
These findings corroborate a HarnessTax study conducted by researchers at UC Berkeley’s Sky Computing Lab and Arena. The Berkeley team evaluated seven models across Claude Code, OpenAI’s Codex CLI, and Pi, a minimal open-source harness with just four basic tools. They tested 30 tasks from each of two benchmarks, repeating each task three times, and found that harness choice shifted pass rates by only a few percentage points while multiplying inference costs up to fivefold.
The Berkeley researchers found that the cost difference can begin with the first model call. Across all seven models, Claude Code's initial context averaged more than 10 times the size of Pi's, largely because of its longer instructions and tool schemas.
Berkeley's HarnessTax showed that more elaborate harnesses produced little improvement in average task success rates on the benchmarks it tested. The new study takes that finding a step further by testing Qwen's 27-billion- and 35-billion-parameter models on a larger task set and measuring harness differences against ordinary run-to-run variation. As the Shandong researchers note of Qwen3.6-35B-A3B, the lowest-scoring model they tested, “The weakest model is not the most harness-sensitive.”
Practical takeaways for enterprise builders
That cost concern extends beyond coding agents. In VentureBeat Intelligence's September VB Pulse survey, 23 of 107 enterprises using a listed agent orchestration platform (21%) had no automated way to stop agents' token spending.
The research offers several practical lessons for engineering leaders and developers building AI applications or allocating budgets for coding tools:
Read vendor benchmarks against the noise floor: Because the result on roughly one in eight tasks can flip across identical runs, a 45-task evaluation suite can only catch a 13-point gap half the time, and even 447 tasks can only resolve a 5.2-point difference at the task-level variation the researchers observed. Builders should treat claims of 2- to 3-point harness gains from single-run comparisons as unproven and run their own baseline configuration twice, as the researchers recommend, before adopting a more complex scaffold.
Strip down turn-one preambles and tool schemas: The system prompt and tool schemas are sent with every step, although cached tokens may be billed at lower rates. When the researchers removed Claude Code’s file-editing tools so Claude Opus 5 had to edit files through bash, its success rate remained within the range of the control runs. Teams should test whether using lean command-line tools instead of dozens of static tool schemas reduces costs without hurting task completion rates.
Tune thinking budgets to the model-harness pair: Claude Opus 5, running on a year-old release of Claude Code, which left extended thinking off by default, solved 34 of 45 tasks, compared with 35 to 36 for the newer release, at 44% of its average estimated API cost (73 cents versus approximately $1.65 per trial). However, teams cannot assume reasoning budgets can be toggled without consequence. For example, turning thinking off for Qwen3.6 caused mini-SWE-agent’s pass rate to drop by 11 points while producing changes in Claude Code and OpenCode that fell within run-to-run variation. The results suggest that the effects of changing reasoning budgets depend on the model-harness combination, not just the model. Teams should benchmark their exact stack before adjusting thinking parameters to cut costs.
“The numbers to carry between vendors are the first call, the growth per step, and the step count,” the researchers conclude.
Want to get more out of your agents with fewer tokens and less context? Apply to attend VentureBeat’s Agentic Infrastructure Stack Summit, October 28 in San Francisco.