Independent field evaluation · TypeSafe AI
Jev 1.13.0: where fast semantic decisions actually help
Our strongest result was not that Jev could replace reasoning. It was that, after deterministic machinery had reduced a messy request to one small semantic decision, Jev was unusually effective.
Practical conclusion: use Jev for bounded, closed-choice semantic residue. Do not use it where deterministic code already knows the answer, for open-ended generation, or to replace a human authority decision.
What we actually tested
We assembled 30 historically grounded cases from a real automation environment. Each case already had a settled label. The public report deliberately omits identifying payloads and internal project names, but preserves the task structure, labels and aggregate measurements.
| Task slice | Correct | Mean probability on settled answer |
|---|---|---|
| Role / stratum classification | 10 / 10 | 0.958 |
| Semantic work vs mechanical/system work | 10 / 10 | 0.986 |
| Evaluation-class assignment | 9 / 10 | 0.740 |
| Overall | 29 / 30 (96.7%) | 0.895 |
The important split: semantic frontier vs known machinery
| Decision posture before Jev | Cases | Correct | Mean probability |
|---|---|---|---|
| Already mechanically decidable | 7 | 6 / 7 | 0.634 |
| Needs semantic intake, then becomes typed | 7 | 7 / 7 | 1.000 |
| Ultimately requires human authority | 3 | 3 / 3 classification | 0.987 |
| Genuine bounded free-text semantic residue | 13 | 13 / 13 | 0.957 |
This is why the headline accuracy is less interesting than the shape of the errors. On decisions that were already mechanically decidable, Jev added uncertainty. On the genuinely semantic slice, it was both accurate and confident.
The one mismatch
One mechanical-compliance case was assigned to the wrong evaluation class. Importantly, Jev's confidence in that selected class was only 0.09, and the returned distribution was split across alternatives. Across the 30-case tranche we saw no high-confidence wrong answers.
That does not prove universal calibration. It does suggest a useful engineering pattern: treat low confidence as a reason to fall back or escalate, rather than forcing every Jev answer into an action.
Latency and usage
Across the 30 provider calls, mean measured client latency was 641 ms, median 637 ms, observed range 573–764 ms, and approximate p95 711 ms. The tranche used 13,029 input tokens and 1,422 output tokens. These are our observed end-to-end client timings, not vendor claims.
Where we would use Jev 1.13.0
- Closed-choice classification after deterministic pre-processing has removed obvious cases.
- Turning unstructured intake into a typed route, after which normal code takes over.
- Small semantic distinctions that are awkward to encode as brittle keyword rules.
- Advisory triage that can safely fall back when confidence is low or the service is unavailable.
Where we would not use it
- Exact arithmetic, state checks, hashes, schema validation or known compliance predicates.
- Open-ended synthesis, planning or generation.
- A final subjective approval that belongs to a human.
- As an architectural dependency: the surrounding system should remain useful when Jev is absent.
A second, smaller check
In a separate five-case replay of previously unseen operational incidents, a bounded Jev classifier matched the subsequently known labels 5/5. We treat that as supporting evidence, not a statistically independent proof of general performance.
Version boundary
This report applies to the model identifier returned by the API during our tests: jev-1.13.0. A later Jev release may behave differently. If thresholds or routing decisions depend on probability values, pin the model version and re-test before moving those thresholds to a new release.
Sources and reproducibility notes
- TypeSafe AI: Introducing System One Models & Jev
- TypeSafe API reference
- Machine-readable version of our measurements
jev-1.13.0, tested 2026-09-19. Do not generalise these measurements to later Jev versions without a new test.Published 23 September 2026 · The Amateur Limited · All research reports · Home