Jev Explained: A Decision Model vs an LLM

Before you start#

This is the first lesson in the AI Models module, and it stands alone. It assumes three things:

• You know what an LLM is, and that it writes its answer one token (a word or part of a word) at a time.
• You've seen structured output: asking an LLM to reply in JSON so code can read the answer.
• You're comfortable with a probability or confidence score: a number from 0 to 1 for how sure something is.

If you want a refresher on checking whether a model's answers are right, the LLM Evals lessons cover it, especially LLM Eval Methods.

Two questions for every bike listing#

Alex runs a small second-hand bike marketplace. Before any listing goes live, Alex needs two answers about it: which category it belongs in (road, mountain, kids or parts) and whether it looks like a scam. Thousands of listings arrive a day, and sellers expect to see theirs live within seconds.

Alex's first version asks an LLM. A seller posts "Child seat for a rear rack, fits most bikes, £35", and the LLM writes back: "This is a child seat, so the kids category fits best, and nothing about it suggests a scam." With structured output it can return {"category": "kids", "scam": "no"} instead. Either way it works, but each answer takes a while to write, every token costs money, and there's no reliable number telling Alex how sure the model was. Alex can't tell the easy listings from the doubtful ones.

What Alex really wants is not a writer. They want something that looks at the listing, picks one answer from their list, and says how sure it is.

What a decision model is#

A decision model is handed three things: a question, a fixed list of allowed answers, and the context to decide on. It returns one of those answers plus a confidence score. It doesn't write a sentence for you to parse: what comes back is one of your own labels, which code can use as it is.

Jev, launched in September 2026 by TypeSafe AI, is a worked example. TypeSafe calls it a "System One model", a nod to the fast, intuitive kind of thinking psychologists call System 1: quick judgements, not long reasoning.

How Jev works: known, claimed and guessed#

Much of what's said about Jev comes from TypeSafe itself, so it helps to sort every claim by how sure we can be of it.

RLCD stands for Reinforcement Learning for Calibrated Decisions. Its goal is the most useful property here: calibration. A calibrated model's confidence means what it says. Take every listing it scored around 0.8: about 80% of those answers should be right. If only 60% are, the score is flattering itself, and a threshold built on it lets mistakes through.

That's TypeSafe's claim. Alex can test it herself: label a few hundred past listings by hand, group the model's answers by confidence, and check the hit rate in each group.

Six listings, one threshold#

Here's one morning's sample. Alex asks both questions about every listing and uses one rule: if the lower of the two confidences is under 0.9, a person checks it. Otherwise a scam is blocked and everything else is published.

The code below runs that rule. decide is a stub standing in for a real decision-model API: it looks answers up in a table, so the sketch runs with no network. Notice the check that the answer is one of the options: that's the guarantee you're paying for.

# A stand-in for a decision model's API: no network, the answers are canned.
CATEGORY = "Which category is this listing?"
SCAM = "Does this listing look like a scam?"

CANNED = {
    "Carbon road bike, 54 cm": {CATEGORY: ("road", 0.97), SCAM: ("no", 0.95)},
    "Kids' 16-inch bike": {CATEGORY: ("kids", 0.98), SCAM: ("no", 0.97)},
    "New e-MTB, bank transfer only": {CATEGORY: ("mountain", 0.93), SCAM: ("yes", 0.96)},
    "Pair of 700c wheels": {CATEGORY: ("parts", 0.94), SCAM: ("no", 0.92)},
    "Vintage frame, price on WhatsApp": {CATEGORY: ("parts", 0.58), SCAM: ("no", 0.71)},
    "Child seat for a rear rack": {CATEGORY: ("kids", 0.94), SCAM: ("no", 0.97)},
}

def decide(question, options, context):
    """Return one of `options` and a confidence. Here: a lookup, not a model."""
    answer, confidence = CANNED[context][question]
    assert answer in options  # it can only pick from the list
    return answer, confidence

THRESHOLD = 0.9
for listing in CANNED:
    category, c1 = decide(CATEGORY, ["road", "mountain", "kids", "parts"], listing)
    scam, c2 = decide(SCAM, ["yes", "no"], listing)
    if min(c1, c2) < THRESHOLD:
        action = "review"  # a person looks at it
    elif scam == "yes":
        action = "block"
    else:
        action = "publish"
    print(f"{action:<8} {listing:<33} {category:<9} scam={scam} ({min(c1, c2):.2f})")

Now a back-of-the-envelope for cost. Assume 50,000 listings a day, 300 input tokens each, both questions in one call. The decision-model price is TypeSafe's published figure; the LLM price is an assumption for comparison, not a specific model.

TypeSafe reports about 70 to 500 milliseconds per decision. Its headline figures are up to about 193 times faster and about 444 times cheaper than LLMs, best-case results from its own internal tests; other coverage quotes different ranges, and there's no independent evaluation yet. Measure on your own traffic before trusting any of them.

Decision model or LLM?#

Start from Alex's budget. Sellers expect a listing to go live in about a second, and Alex pays for every check. An LLM with structured output gets the labels right too, but it spends time and tokens writing them, and it leaves Alex no trustworthy number for sorting sure answers from shaky ones. A decision model promises the opposite on all three: a vendor-quoted fraction of a second, a price low enough to check every listing, edit and message instead of a sample, and a score Alex can turn straight into a routing rule.

Good fits are small, frequent decisions with a known list of answers:

• Checks on everything: scam, spam and policy questions on every listing, edit and buyer message, not on a sample.
• Free text into fields: a seller's description becomes a condition label (new, good or worn) the search filters can use.
• One step at a time: a large model plans a job such as re-pricing stale listings, and a cheap decision checks each step (is this comparable listing a close match?).
• Many decisions a second: TypeSafe's own demo has Jev playing the game Doom at about ten decisions a second.

Keep the LLM when you need written text, an explanation, open-ended answers or multi-step reasoning. And know the downsides:

• No independent benchmarks yet: the speed, cost and calibration claims are the vendor's.
• Text only, and no web search: it decides only on what you pass it. If the scam signal is "this seller was banned last week", put that fact in the context.
• No explanation, just an answer and a probability. Alex can't show a seller why their listing was held.
• Reported weak at counting: do exact arithmetic and comparisons in code, so "is the price below 20% of retail?" never reaches the model.
• Security is contested: one security vendor, HiddenLayer, says Jev is more open to some attacks than LLMs are; TypeSafe hadn't responded when the report came out. Treat it as one vendor's claim, and test your own inputs.
• Early access, a single vendor, and no published weights or architecture details: pricing and behaviour can change.

Expert corner, and where this may go#

The real design choice is the threshold. Raise it to 0.95 and fewer mistakes are published, but more listings wait for a person. Lower it to 0.8 and the queue shrinks while errors grow. Pick it from your own calibration check, and set it per question: a wrong category costs little, a missed scam costs a buyer their money, so the scam question deserves a stricter bar.

At scale, watch for drift. New scam patterns change what listings look like, and confidence that was calibrated last month may not be today. Keep sampling auto-published listings for human review, and track the hit rate per confidence band over time.

In a system design interview, "design moderation for a marketplace with a million listings a day" fits this shape: a cheap classifier on every item, a confidence threshold, a human queue for the uncertain band, and a bigger model only where you need explanations. Mention that the threshold sets the size of the human queue, and you can estimate that queue from the share of listings below it.

One sentence to remember: an LLM writes an answer, a decision model picks one from your list with a confidence, and that confidence is what lets you act on every event automatically.

Practice

Check your understanding

FAQ

Frequently asked questions