Generative AI is an incredible business advance, giving companies the opportunity to handle a wide variety of knowledge work tasks with ever-changing, inconsistent input data. But what about ensuring the outputs remain consistent and are in-line with the business goals? That's a much harder task, given, by the fundamental way in which it is designed, gen AI is "non-deterministic" — that is, outputs are constructed on the fly and may vary wildly even with the same or similar input information.

In order to help businesses harness the tech and make it more consistent, the stealthy startup CTGT is releasing a new system called Concordance under the Apache 2.0 open-source license, giving enterprises free tooling to investigate why an AI application makes different decisions when asked to perform the same task.

The release includes the testing protocol, reference implementation and experimental data, according to CTGT. Businesses can use and modify the software commercially under the license; the company says it has no paid or hosted Concordance edition planned.

The tool addresses a problem CTGT encountered in its own compliance application: processing the same banking manual four times produced four different sets of obligations. One run omitted a requirement to escalate a serious issue to the compliance chief—even though the underlying model responded consistently whenever it received an identical request.

For enterprises moving from AI assistants to agents that approve, flag or record consequential decisions, Concordance offers a way to locate where those discrepancies begin.

“You have to start where the AI actually commits something that someone could later ask you to defend,” CTGT founder and CEO Cyril Gorlla told VentureBeat in an interview.

Why the same document produced different decisions

CTGT describes itself as an applied interpretability lab: it studies how AI models and their surrounding software behave in settings such as banking and insurance. Gorlla said he developed the company from research following his studies at UC San Diego and work at Stanford. Its commercial business includes a policy enforcement engine for regulated enterprises, separate from Concordance.

The initial experiment examined CTGT’s own application, rather than a disclosed customer deployment. A technical note details how the system processed a nine-page retail banking manual using Qwen3.8-27B with SGLang’s deterministic inference mode.

Four runs extracted 32, 31, 33 and 33 obligations. Matching totals did not mean matching results: all four outputs differed.

Yet the model returned consistent responses across all 74 identical requests that recurred during those runs. The application kept changing what it sent.

“So even if you fix non-determinism in the model, your pipeline can still give you four different answers,” Gorlla said.

CTGT identified three causes. Upload-specific identifiers appeared inside prompts; a 30-item routing list changed order; and a topic registry expanded between runs, altering subsequent requests.

One comparison showed differences in just four identifier lines within a 12,525-character request. The model then classified five conflict-resolution rules as background information, omitting an obligation another run retained.

CTGT replaced the identifiers with stable labels, sorted the list and isolated each test in a fresh organizational environment. Five subsequent runs each produced the same 34 obligations.

Comparing the stored results required normalizing upload-specific identifiers, although intermediate parsing and interpretation outputs matched without that adjustment.

The experiment demonstrates a failure in one application processing one document. It does not establish how common the problem is across enterprise deployments.

What Concordance measures—and what it does not

Concordance examines both the software serving a model and the application using it. It captures exact model requests and responses, compares outputs at successive stages and identifies the earliest divergence.

That separates two debugging problems: a model returning different responses to identical requests, and application code altering the requests themselves.

Thinking Machines Lab’s research helps explain the first problem. Even the least-random generation setting does not guarantee identical answers, because changes in server batching can affect numerical calculations. CTGT uses SGLang’s deterministic mode, which its representatives said builds on that research.

In CTGT’s separate serving test, Qwen3.8-27B matched its control response in all 800 requests under the deterministic configuration. That measures exact textual agreement, not whether the answers were correct.

The distinction becomes stark in the compliance experiment. Replaying the request behind an incorrect classification produced the same mistake eight times.

“So I think the thing that a lot of people conflate is that repeatable isn't necessarily the same as right,” Gorlla said.

CTGT identifies assessing whether decisions remain faithful to policies despite changes in wording as its next measurement challenge. Concordance’s reported results do not establish that it automatically prevents policy violations.

Could decision models such as Jev prevent the problem?

The rise of decision models offers enterprises another way to reduce some of these risks at the design stage. TypeSafe’s System One approach, embodied in Jev, returns constrained choices, scores and probabilities for software to consume. Developers specify the available answers instead of asking a model to generate an unrestricted response. AWS also describes a broader decision-model category that includes Jev, SemIf and Strands Decider 2B.

TypeSafe’s recommended architecture leaves workflow control and deterministic business rules in code, using AI for narrower judgments about unstructured information. Applied to a compliance workflow, that could mean a model classifies a conflict while code requires escalation whenever the relevant condition holds. A required step would no longer depend on an agent choosing to execute it. This is an architectural implication, not a configuration CTGT tested.

That approach addresses part of the same reliability problem, but at a different layer. A decision model makes a judgment; Concordance tests whether the larger system reproduces its judgments and where discrepancies begin. Constraining the answer format prevents an out-of-schema response, but a classifier can still choose the wrong permitted answer. If an earlier document-processing stage omits an obligation, a downstream classifier cannot reliably enforce a rule it never receives.

TypeSafe’s own limitations for Jev 1.13 include sensitivity to the ordering of choices and difficulty with irrelevant context. Its self-consistency demonstration reports label changes on two of eight questions across repeated tests that included a fresh identifier in each request. Routing uncertain outcomes to human review improved agreement, but TypeSafe explicitly cautions that this does not make the model deterministic. Because those requests changed, the test does not establish instability under identical inputs; it illustrates why stable task meaning alone is insufficient.

For buyers, the approaches can therefore complement each other: use bounded model judgments and code-enforced rules to reduce opportunities for failure, then test the complete workflow for repeatability and accuracy. CTGT has not provided a Concordance comparison using Jev, so the available evidence does not establish which architecture would perform better on its banking example.

How it fits alongside existing monitoring tools

Gorlla positions Concordance as an addition to existing tracing systems, which record the steps an AI application takes.

“And there's nothing to rip out, right? It's free. It's Apache 2.0,” he said.

That positioning matters because competing platforms already offer substantial debugging capabilities.

Concordance’s proposed distinction is the focused comparison procedure: repeat the work, align the stages and determine whether changing inputs or changing responses explain the divergence. Its value would lie in making that investigation easier to perform and reproduce.

CTGT supplied no head-to-head benchmark proving that these platforms cannot uncover the same failures. Their documented features overlap with the evidence Concordance needs, supporting Gorlla’s argument for using them together.

Gorlla also described an unnamed consulting firm finding overlooked false positives using CTGT’s earlier LineageEval framework. That anecdote concerns a different tool; it does not establish customer results for Concordance.

Where enterprises should start

Gorlla recommends prioritizing decisions with operational consequences, including approvals, flags, compliance obligations and insurance claims. A drafting assistant whose work receives human review presents a different testing priority from an agent whose output becomes an authoritative business record.

“You don't need to monitor everything all the time. What you need to be able to do is replay any decision and get the same answer, right?” he said.

His recommendation is to preserve complete requests and responses for model calls that commit decisions. Teams should repeat tests when models, prompts or source documents change, and maintain trusted examples with known expected results to check whether behavior has deteriorated.

The approach does not require access to a model’s internal reasoning, Gorlla said. Enterprises using proprietary APIs can compare inputs, outputs and decisions, while access to model weights can support deeper investigation into why a failure occurs.

For buyers, model choice, workflow design, reproducibility testing and policy enforcement address different failure modes. Decision models can constrain individual judgments, code can enforce mandatory actions, and Concordance can help locate variation across the workflow. Teams still need independent checks against their policies to establish whether a repeatable decision is correct.

CTGT directs developers to its Concordance research page. The proprietary compliance application used in the experiment is outside the open-source release. Enterprises adopting the tooling would still need to integrate it with their workflows and account for the model calls, storage and engineering work involved in testing.