A cheap model watching an expensive one

1What Jev is, from zero

You do not need to have read part one.¹ Here is the whole idea again.

A classifier is a program that sorts things into boxes. Your spam filter is one. It reads an email and picks "spam" or "not spam". It does not write you an essay about the email. It picks.

Jev is a classifier released by TypeSafe on 15 September 2026.² You give it two things:

• The state. Any text. A message, a page, a command someone is about to run.
• The questions. Each question comes with its own fixed list of allowed answers, which you write.

It gives back the answer it picked, a probability for each option, and a confidence number. Nothing to read, nothing to parse.

Why a thing that only picks is useful

A chatbot style model writes its answer one small piece at a time. Even to say "yes" it reads your whole question, then writes. Often it writes a paragraph of reasoning first. Then your software has to find the answer inside the sentence.

Jev skips all of that. Three things follow, and each one matters for a guardrail:

• It is fast. Ours answer in about two thirds of a second, in front of every action.
• It is cheap. Cheap enough that nobody has to ration the checks.
• It cannot invent an option. It can only return an answer from your list. TypeSafe is careful here: that is a guarantee of the format, not a proof that the pick is right.²

It can still pick the wrong answer from your list. That is what the confidence number is for, and most of this note is about living with that honestly.

2Why a watcher beats a self check

What we are guarding

We run AI agents that do real work over many steps. They read files, run commands, write plans, build things. Around them we run hooks.

A hook is a small program that fires at a fixed moment. Not when the agent decides to call it: at a moment the surrounding software owns. Three moments matter:

• When the user sends a message.
• Before the agent runs a tool.
• When the agent finishes its turn.

Each hook asks one question. If the answer is bad it blocks the action, or it leaves the agent a line to read.

The old way, and why it failed

Almost all of ours were fixed rules. They counted words, matched keywords, measured length. They are free and instant, and they are blind, because every question they ask is about meaning.

Three examples from our own live system, all measured:

• The end of turn nudge counted words and keywords, and raised 82 wrong alarms in 104 turns.
• The card gate, which stops the agent asking the user a pointless question, was wrong on 10 of the 11 times it fired.
• The close check, which is supposed to confirm that a finished task was really finished, accepted any eight characters as proof.

A wrong alarm is not free. It burns a whole turn of the agent's work and teaches it to ignore the next alarm.

Why not just ask the big model

The obvious fix is to ask a large model to review the work. We tried exactly that, on the same 469 test cases, and published the numbers in part one.¹ Three things stopped us:

• It is slow. The small model took about 6.0 seconds per check, the mid size one about 6.3. Jev took 0.66.
• It is expensive per check. Per answer, Jev cost 79 times less than the small model and 214 times less than the mid size one.
• It was less accurate where it acted. When it decided, Jev made 3.8 to 7.5 times fewer mistakes than the large models did.

There is a fourth reason, and it has nothing to do with price. A guard should not be the thing it is guarding. When the same model both writes the reply and grades it, you have one opinion, asked twice.

3How to wire it to any agent

This section is the practical one. None of it is specific to our assistant.

Rule 1: one call per moment, not one per check

Jev answers every question in a call at the same time. The call costs the same as its slowest question, not the sum of all of them. Ten checks asking separately is ten waits. The same ten riding one call is one wait.

So the unit is not the check. The unit is the moment.

The arithmetic that follows is the whole point:

• A turn that only reads and answers: one call.
• A turn that imports task rows and replies: two.
• Nothing costs three unless it both gates a tool and closes rows.

Rule 2: drop the questions that cannot apply

Before the call is built, plain code throws out every question whose subject does not exist this turn.

• No board write this turn, so the log question is not asked.
• The reply names no finished work, so the claim question is not asked.
• The message is under twelve words, so the instruction questions are not asked.

A call ends up carrying between one and four questions. Never ten.

Rule 3: code overrules the model on facts code already holds

The model is asked about meaning only. Anything the surrounding software already knows is applied after the answer comes back, and it wins.

• A card was already shown to the user, so no "you should ask" nudge fires.
• A helper process is running, so the "nothing will do this" nudge is wrong by construction.
• The reply is a bare token the user asked for, so the readability question is dropped.

This is not a performance trick. It removes whole classes of error from the model's job.

Rule 4: fix the order of decisions before you ship any of them

One call comes back with several answers at once. They will contradict each other. Decide the order once, in one place:

1. A block beats a nudge. When a gating check refuses an action, nothing else speaks. The action did not happen, so the rest is moot.
2. A code fact beats a judgement. See rule 3.
3. One voice per turn end. When three questions all want to say something, say the most expensive one and record the rest.

That last one is not theory. Our measured failure was three nudges in one message, which teaches an agent to read none of them.

Rule 5: fail open, and mean it

Every check has a plain fallback: the rule it runs today. The client gives up after 3 seconds and returns nothing. No key, no network, no answer, a malformed answer: all the same path.

# jev_checks.py, the shipped thresholds.# Each number was picked on a tuning split, never on the set its error rate comes from.THRESHOLDS = {"CLAIMDONE":  {"claim": 0.70, "explain": 0.50},   # 0.0% of 299"CARDGATE":   {"fire": 0.875},                    # 0.0% of 146"PLAINSCORE": {"fire_at_or_below": 1.5},          # 3.7% of 242"PROMISE":    {"fire": 0.375},                    # 4.0% of 347"DIRECTIVE":  {"fire": 0.75},                     # 4.1% of 392"DIARYFIT":   {"fire": 0.825},                    # 4.3% of 188}

Two habits make failing open safe rather than sloppy:

• Every threshold carries its measured error rate, in the file. A number with no rate beside it is a guess someone will inherit.
• Tell "silent" apart from "no answer". A check that said nothing and a check that never ran look identical in a log, and confusing the two silenced one of ours outright.

4What cost us the most

These are in order of what they cost us, not in order of how clever they sound.

4.1 A judge and its answer key must carry the same definition, word for word

An answer key is the set of correct answers you score against. Ours is written by two independent AI reviewers, plus a third to settle disagreements.

The trap is that the key and the question are written at different times, by different hands, in slightly different words. Then you spend days tuning the question, and the question was never the problem.

• One check reads whether an edit quietly deleted a rule. It was wrong on 24 cases. Twenty of those 24 were one contradiction. The judge's wording said a list of fields is not a rule. The key said a value the reader must use is a rule.
• Writing the same list of what counts, into both, took that check from 10.6% to 4.3% on cases the wording had never seen.

The same thing again, on a different check, with the same shape:

• Our close check scored 28.8% against one answer key and 7.4% against another. Same judge, same wording, same cases.
• The only change was that the owner ruled on what the check is for, and the key was re labelled to match.
• Agreement between the two independent reviewers moved from 83% to 95.4% at the same time. That is the tell: when the two people writing the key cannot agree, the key is the problem.

4.2 Split the subject. Never split the judgement

The intuition that small questions beat big ones is half right, and the half that is wrong is expensive.

• Splitting the subject works. "Which line changed?" gives the model one small thing to look at.
• Splitting the judgement fails. "Is this part proven?" asked six times is the same hard question asked six times, and you take the worst of six answers.

Measured on the close check, on 500 fresh cases:

It was measured twice. On a second check, the per line design scored 17.9% against 10.6% for one question over the whole change.

Why it fails is worth knowing. Asked "is this small part shown?", the model answers "touched" at 0.8 to 0.95 for parts the proof never mentions at all. A fast classifier is the wrong tool for an exhaustive audit.

4.3 Where two reviewers sit on the fence, no wording can win

Every case in our key carries two independent labels with a probability each. When one of those probabilities sits near the middle, the case is not hard for the model. It is undecided by the rule.

On one check:

• 42% of cases had at least one reviewer sitting between 0.35 and 0.65.
• Over the whole set the judge reads 7.8% wrong.
• On the 215 cases both reviewers were confident about, the judge is wrong on 2, which is 0.9%, with zero wrong alarms.

Neither number is a lie. The first one measures the key's own noise as well as the judge. So every rate we publish now names the confident set, and names how many cases were dropped to get it.

4.4 More rule text made it worse. One test beat six shapes

Our hardest check asks whether an incoming message puts new work on the plate. We wrote rules for it, then more rules, then exceptions to the rules.

Then the owner ruled on real messages by hand. 39 of them, then 11 more.

• Every one of the first 39 rulings contradicted the key the old error rate was measured against. The rate was not a judge's error rate at all.
• Six carefully written shapes collapsed into one test. A message is a new item only if it carries multi step work. The six were a go ahead, a follow up, an idea posed as a question, and three more like them.
• Two independently worded reviewer passes now match the owner on 38 of the 50, and the judge reads 4.6%.
• 12 of the 50 are still missed. That is the honest remainder, and it is written down rather than tuned away.

One other thing the model does that no amount of text fixes. Jev follows the broad principle over the specific example. A sweeping first rule overrides an exception written further down. Put the narrow exceptions first, and keep the general rule narrow.

4.5 A guard that always allows is invisible in every log

This is the cheapest mistake to make and the hardest to see.

One of our checks never fired. Not once. It was not too strict and it was not badly worded. The code reading its answers asked for fields the shipped judge does not send. So every probability it read was 0.0, and the check returned "allow" on every single edit.

It happened twice, in two different checks.

• The second one read probability where the answer arrives under noul. It looked broken for three rounds of tuning, and 35 wrong alarms were entirely the runner's fault, not the model's.

The fix is one line of discipline: print one raw answer before you treat any error as a judging error. A silent guard looks exactly like a guard with nothing to complain about.

4.6 Two smaller ones, both worth the minute

• Give the model the speaker's vocabulary. Our team says "lanes", "battery running", "next on my side". Jev does not know a team's private words. A short glossary inside the question took one check's errors from 26 to 15.
• Score the action, not the label. Our key labels eight kinds of reply, but the hook only does three things. Scoring the eight labels showed 20 errors on fresh turns. Scoring the three actions showed 7. Two different labels can mean the same action, and then the label is wrong and the hook is right.

5Results

Twelve checks. Each measured on real cases its wording had never seen, against a key labelled by two independent reviewers plus a resolver.

The same twelve, against what runs today

What it buys

• Four checks did not exist before. The promise, the standing instruction, the agent complaint, and clarity read as a reading rather than a word count. They cost nothing extra, because they ride calls the other checks were already making.
• Three checks were wrong more often than right. The stop nudge, the card gate and the doc loss rule. All three are now at zero on cases they had never seen.
• Two checks were letting everything through. The close check took any eight characters as proof. The completion claim fired on every finish word without reading what followed.

What it costs

• One call per moment. 655 ms at the median on a Mac, 784 ms at the ninth decile, 992 ms at the worst of twelve.
• A turn that reads and answers pays one call. A turn that imports rows and replies pays two.
• With no key, no network or no answer, every check returns to exactly the rule it runs today.

What these numbers do not prove

Four things bound everything above, and they are worth more than the table.

• A rate is reported on the cases the key is sure about. Where the two reviewer passes disagree, or one sits near the fence, the case is dropped and the drop is named. Row shape dropped 184 of 399. The close check dropped 109 of 500. Doc loss dropped 41 of 151.
• Zero is not the same as proven. Doc loss made no mistakes, on 50 cases. Fifty cases put the true rate anywhere under 7.1%. More cases, not more wording, is what closes that gap.
• One check rests on 50 rulings, not on a rule anyone had written down. The request check's answer key is the owner's own hand rulings. 12 of those 50 are still missed.
• One company, one assistant. These are our hooks, our questions and our agents. Another system will have other questions, and will have to measure them.

6What others are building

We are not alone in this, and the nearest public example is worth an honest paragraph.

Foreman is an open source project on GitHub, MIT licensed.³ It is 38 files and about 110 KB. We read it whole on 17 September 2026. It runs a coding agent as a subprocess, and at the same time runs a second loop that watches it. Every few seconds the watcher packs what it can see into one bounded snapshot. Then it asks a fast model nine yes or no questions about it. The model returns nine probabilities. A short piece of plain Python then picks one of seven actions. Everything seen and decided is appended to a file, so a run can be replayed.

The shape is the same as ours. Two things in it are better than ours, and we are taking both:

• One ordered decision table, with every threshold beside it. Its whole control surface is 102 lines in one file, read top to bottom. Ours is spread across several adapters, and we have a decision log showing one advisory silently losing its slot to another.
• One bounded snapshot, built once per decision. It truncates every field and writes down how much it cut, so a truncation is never silent. Our checks each gather their own evidence, with their own caps or none.

One thing in it we will not copy. It can stop a live worker when one probability crosses a fixed number, and its own README says false positives can stop useful workers. Its own documentation also lists what the project does not establish, including that those default numbers are calibrated. That is an honest boundary. It is the same boundary we drew, the other way round. In our layer a judgement never stops work on its own.

7The fine print

• The answer keys are mostly AI labelled. Two independent reviewers plus a resolver, with one exception: the request check, whose key is the owner's own hand rulings.
• Thresholds were picked on a tuning split. No threshold in this note was chosen by looking at the set its error rate comes from.
• Latency was measured on one laptop. 655 ms median, 784 ms at the ninth decile, 992 ms at the worst of twelve, one call at a time.
• We quote no price here. Part one carries the measured cost per answer for our checks.¹
• TypeSafe's claims are TypeSafe's. Section 1 describes its model from its own documentation. We did not rerun its benchmarks.
• Foreman's design is read from its source. We read the repository; we did not run it in production.

8Glossary

95% bounds

The range the true error rate probably sits in. Narrow means many cases. Wide means few.

Answer key

The correct answers for a set of test cases, written independently, used to score a check.

Classifier

A program that sorts things into boxes, like "spam" or "not spam".

Confidence

A number saying how clear cut a question was for the model.

Fail open

When the check cannot get an answer, it does exactly what it did before the check existed.

Guardrail

A check that sits between an AI agent and an action, and can stop it or correct it.

Held out set

Cases kept aside and never used to improve a check, so the score is honest.

Hook

A small program that runs automatically at a fixed moment around an AI agent.

Labelling pass

One reviewer going through every case and writing the correct answer, without seeing what the system said.

Median, ninth decile

Half of the answers were faster than the median. Nine in ten were faster than the ninth decile.

State

The text the classifier reads: everything it needs to know about the situation.

Wrong alarm

A check that fires when it should not. It costs a whole turn of the agent's work.