Jev returned valid answers. The routing still needed fixing.

By Sandeep · Evaluation conducted 19 September 2026

My verdict: Jev’s opportunity is not “more intelligence.” It is making narrow semantic judgments cheap and fast enough to use routinely inside ordinary software.

Think of it as a promptable classifier and scorer: you supply the evidence, question and possible answers at request time; it returns a bounded judgment for code to use. That flexibility is useful. It does not replace deeper reasoning or give the model authority to act.

Our tests put Jev on the shortlist for that role. They do not establish it as the best purchase against competing models. I would compare cost per acceptable decision, including mistakes, escalation and human review, rather than divide an accuracy score by the API price.

I asked Hermes Agent to test TypeSafe’s Jev against its own playbooks. I wanted actual requests and recorded answers, including cases where the application should refuse to make a decision.

One result explains why.

The input was “Explain what a monad is”. A shopping-support routing recipe assigned it to product_question, with confidence of 0.99.

The response had the right structure. It selected an allowed category. It also cleared the recipe’s confidence threshold for avoiding review.

There was one problem: this was not a shopping-support request.

That is the part of building with AI I wanted to examine. Getting a model to return something software can consume is useful. Deciding whether the software should act on it is another job.

A decision component, rather than another chatbot

With Jev, you supply the evidence, a question and the possible answers. It returns a choice, a probability that a condition holds, or a score against defined levels. Your application uses that result.

That makes it interesting for narrow jobs such as routing requests or checking whether a passage supports a claim. You do not necessarily need a paragraph of generated prose to answer either question.

The evaluation made 190 live requests to the pinned jev-1.13.0 model, plus one request to list models. All 190 evaluation responses passed the tested checks for answer keys, types, ranges and option membership. There were two probability maps whose values summed to 0.99, a numerical wrinkle consistent with rounding rather than an invented label.

The estimated TypeSafe inference charge was $0.05307 at the test-date pricing. That excludes Hermes, other tools and producing the reports. It is a usage estimate, not a checked invoice.

Those are useful results. They do not mean 190 independent problems were solved correctly: the diagnostic contained 58 primary cases, and many requests were repetitions or controls.

The missing answer was “none of these”

The original routing recipe handled all twelve in-scope support messages as intended. It also assigned four unrelated messages to product_question. A bedtime-story request was among them.

But the model had not been offered a no-match category. Blaming it for failing to select an unavailable answer would miss the design problem.

The recipe assumed that uncertainty over the available categories would catch requests that did not belong in any category. These probes showed why I would not rely on that assumption.

A subsequent development check added a no-match option and a separate question about whether the message belonged in shopping support. It kept the four unrelated requests out of automatic routing and sent them for review, while retaining four straightforward controls.

That was a repair tested on known examples, not proof of general accuracy. The next step would be an untouched set of representative requests, including less obvious out-of-scope cases.

TypeSafe’s own integration guidance puts the boundary plainly:

Typed output guarantees the interface, not truth.

The surrounding application still has to represent the possibility that no available action is appropriate.

The speed result depended on the comparison

Another test asked thirteen questions about the same document.

One batched request averaged 0.861 seconds. Thirteen sequential requests averaged 10.855 seconds. That gives a 12.6× speed advantage for batching.

Then the comparison changed: what if those separate requests ran concurrently? Their mean was 1.020 seconds. The advantage became about 1.18×.

Both figures describe the recorded measurements. Neither is a comparison between Jev and another model.

The input-cost advantage remained 12.25× because the separate requests repeatedly sent the same document. Concurrency changed how long the client waited; it did not remove duplicated input.

This was a small experiment. The concurrent control came later and had fewer repeats, and timings included client and connection overhead. Batched and sequential answers were close, but not perfectly identical. I would use the result to justify testing batching in a similar workload, not to promise a universal speedup.

What a fair comparison needs

TypeSafe’s workflow evaluation uses reference answers derived from GPT-6 Astra and Claude Fable 5.1 at high thinking. Agreement with those references is not independently established correctness. The company also acknowledges that its adapter makes competing LLMs output probabilities, which can be slower and more expensive than returning decisions without them.

Our own labels were fixed before inference, but they were AI-authored and AI-reviewed, not human-adjudicated ground truth. One citation label was debatable. The existing cases are now known development material; a stronger comparison needs fresh held-out cases and independently adjudicated labels.

I would test both the same typed interface and the best practical alternative. If the application only needs a label, requiring every competitor to generate a full probability map can charge it for output the application does not need. Thresholds also need validation for each model and output type. This is a proposed comparison, not a benchmark we have run.

The application I would try first

Citation checking was more interesting to me than giving the model authority over an action.

One test used the JSON Web Token specification. A passage described how to check an expiration claim. The proposed claim went further: every JWT must contain an expiration claim. The specification makes that field optional, and Jev caught the distinction.

That is a useful editorial check. A real quotation does not necessarily support the claim attached to it.

The experiment also caught two fabricated quotations with ordinary string matching, before calling Jev. Those catches belong to the code, not the model.

My next step would be a shadow-mode citation checker over public sources: flag questionable claim-and-passage pairs for an editor, then measure useful catches and unnecessary review work. That integration has not been built or validated in production by this evaluation.

Illustrative support flow, not an observed API result or an implemented production integration. Generated by Hermes Agent using tldraw.

What this means for 11factor

This is a concrete case for keeping evaluation, orchestration and permissions distinct.

A response can satisfy the tool contract and still be inappropriate for the workflow. A confidence score can help decide what to review without granting permission to act. A faster request can still create more downstream work if it sends the wrong cases to the wrong place.

I would start with a bounded judgment, make “no suitable answer” explicit where needed, and test the decisions that reach the application. Exact matching, arithmetic and access checks should stay in code when code can determine the answer.

Jev earned further investigation for that role. The test did not establish production reliability, calibrated confidence or superiority over another model. The small guardrail and skill-selection exercises in the full report are diagnostics, not safety certification or evidence of improved agent performance.

A cheap API call is only useful if it helps the application reach an acceptable decision. That is the cost I would measure next.

Read the two reports

• Jev: useful decisions, not guaranteed answers (PDF) — the hosted API evaluation, methods, findings and limitations behind this article.
• Jev and SemIf: the same interface idea, different evidence (PDF) — a separate companion covering local browser checks and an audit of published result artifacts. It is not a controlled head-to-head benchmark or evidence that SemIf reproduces Jev.

Method and disclosure: I commissioned the work; Hermes Agent executed the tests and assisted with research and writing. Inputs were public or synthetic. An AI assistant authored the cases, and a separate AI review challenged the initial labels without seeing outputs. This was not a human-adjudicated benchmark. Later repair probes were kept separate from the primary results. The full reports retain the limits of both the tests and the independent fact/privacy review.

Primary references: Jev API, typed question primitives, vendor evaluation methodology, vendor comparison caveats, confidence, intent routing, parallel questions, citation checking, model limitations, RFC 7519.

Reformatted by AgentSmith ( Hermes Agent)