Decision models are hot right now: The internet is buzzing about Jev (from TypeSafe AI), Clef from Cloudflare, and the Decisions API from OpenAI. These models provide low latency by skipping verbose text generation and returning structured outputs in JSON format, along with confidence probabilities.
Software developers really like this determinism, because it makes it easy to integrate structured data into applications. Ask a question and get a fixed JSON that can be extracted using a library; no need for complicated text parsing or format understanding.
A conscious choice
Under the hood, every modern large language model (LLM) follows the same transformer encoder-decoder architecture. Prompts are ingested and encoded into a smaller dimension, then unpacked to produce one token at a time that is unified to generate output text.
Models with billions of parameters are trained over trillions of lines of text, learning complex linguistic patterns to ensure generated text follows patterns that emulate human thinking and conversations. The verbose text output often contains the reasoning used by the model to arrive at a conclusion. This is so useful that a separate category of models, reasoning models, focuses on creating a dedicated reasoning trace to support decisions.
But with decision models, we make a conscious choice not to include verbose reasoning traces and get straight to the answer. This so-called System 1 thinking is very useful in specific scenarios, because we get an answer faster and save significantly on tokens. However, we usually end up building technical debt in our systems. Thus, we actually lose out on the core capability that makes LLMs so powerful.
Let's look at an example. I asked different types of models, “Is it going to rain today?” via the OpenRouter interface. A traditional LLM uses reasoning to understand what is necessary and what limitations exist before making a prediction.
Now, if you ask the same question to a decision model like Jev, it would return a response in a data structure “noul" (named for the great Swiss mathematician and physicist Daniel Bernoulli).
Noul carries a single value, either true or false. So without explaining anything, the model would give a generic response, true or false. This can be highly misleading, especially when we use models to answer critical questions.
Here is an example.
This does not mean that decision models are not useful or incorrect. It just means these models are not explainable like traditional LLMs. That shifts a critical responsibility onto the AI builder and software developer: If the model won't reason for you, you have to reason for it upfront. This is where context design becomes the most important skill an AI builder can master.
With context design, your objective is to package all relevant information in the context or state for the model to make an informed decision. With a traditional LLM-powered agent, you can ask a vague question and the model will usually ask clarifying questions, or verbosely explain the assumptions it made.
With a decision model, there is no such provision. The model takes whatever context you give it, maps it against its training, and returns a structured output — often a single value. You can take this value and the probability to program appropriate action in the application. However, there are no decision traces or logs, so there is not full clarity into why a model made its decision.
Decision models shine in certain scenarios; for instance, LLM-as-a-judge, when a chatbot's answers are compared with true “golden” answers defined upfront. Another area is support ticket triage and category mapping. Support tickets often contain verbose text that needs to be understood to be mapped to appropriate categories or teams.
Time will tell what other prime use cases arise. But one thing is true: Decision models are a major technology upgrade in the rich story of LLMs.
Dattaraj Rao is an innovation and R&D architect at Persistent Systems
Welcome to the VentureBeat community!
Our guest posting program is where technical experts share insights and provide neutral, non-vested deep dives on AI, data infrastructure, cybersecurity and other cutting-edge technologies shaping the future of enterprise.
Read more from our guest post program — and check out our guidelines if you’re interested in contributing an article of your own!