calibrated decision models
certo.
Small models that know how sure they are.

A toolkit for building narrow, calibrated decision models — trained on synthetic data with a known answer key, so the confidence they report is trustworthy by construction, not patched on afterward.

pip install certo research preview MIT non-generative · typed probabilities

Read the technical report →

Inspired by Jev / “System-1” decision models. Independent project — not affiliated with TypeSafe.

the problem

The answer is easy. The confidence is the hard part.

A decision model earns its keep when it says “route to billing, 0.82” and the 0.82 is real — so you can escalate on doubt or trade quality against cost. Train on hard labels (the usual way) and the model still picks well but turns overconfident; you’re left patching calibration after the fact.

Train on the distribution

certo trains against the full answer distribution (a proper scoring rule), so probabilities mean what they say — calibrated by construction.

Grade against the truth

Because the data is generated from a known posterior, every model is scored against the exact answer — not just accuracy, but fidelity.

how it works

A known-answer world, dressed in language.

You define a decision as a synthetic world whose answer is computable. certo renders it to natural language, and a small encoder learns to read the state and each option and emit calibrated, typed probabilities. The answer key both trains and grades.

01
Synthetic world
a decision with a known, closed-form answer
02
Rendered to text
state + runtime options, many wordings
03
Small encoder
ModernBERT reads state & options
04
Typed probabilities
choice · binary · score · multi-label
stateEnglish encodercalibrated p exact answer key — trains & grades (the model never sees it)
The model only ever reads the text; the exact answer is used to build the training target and to grade fidelity.
evidence — controlled worlds

Calibration recovered from plain English.

Measured against the exact posterior (KL, lower is better) on held-out wording. Training on the full distribution lands essentially on the answer; single-label targets sit near “ignore the evidence.”

0.004
KL to exact answer
(soft targets, from text)
≈ Bayes
accuracy at the optimal ceiling
0.00
option-order sensitivity
(invariant by construction)
0 0.8 KL to exact answer ignore-evidence baseline 0.605 0.004 soft 0.72 sampled 0.61 argmax
ModernBERT reading English · held-out wording · robust across seeds. And in a routing task, only calibrated probabilities support cost-aware choice — accuracy alone can’t.
quickstart

Define a decision, generate data, ship a classifier.

The interface certo ships (research preview — API stabilizing).

# 1 — describe your decision as a known-answer synthetic world
from certo.synth import EvidenceWorld
world = EvidenceWorld(classes=["billing", "tech", "refunds"], features=12)
data  = world.generate(n=20_000)          # (state, options, EXACT target)

# 2 — train, calibrated by construction (soft/distributional targets)
from certo import train
model = train(data, backbone="ModernBERT-base", target="soft")
model.save("my-router")

# 3 — use it locally: state + runtime options -> calibrated probabilities
from certo import DecisionModel
m = DecisionModel.load("my-router")
r = m.decide(state="card charged twice, refund please",
             options=[{"name":"billing", "description":"charges & payments"},
                      {"name":"tech",    "description":"app problems"}],
             kind="choice")
r.probs        # {"billing": 0.86, "tech": 0.08, "OUT": 0.06}  — calibrated
r.abstain      # True when the top probability is below your threshold
hard tier

When one forward pass isn’t enough, reason — briefly.

Certo-R1: a Qwen3-4B fine-tuned for concise, verified reasoning — behind the same API.

A calibrated encoder is cheap and trustworthy, but hard, reasoning-heavy decisions (multi-step arithmetic, timezone math, nested exceptions) need generation. Certo-R1 reasons in a sentence or two and returns the chosen option — non-inferior to full verbose reasoning at ~15× fewer tokens and ~8–11× lower latency. The natural architecture is a cascade: the calibrated encoder decides the easy majority; low-confidence cases escalate to Certo-R1.

from certo import ReasoningDecisionModel
m = ReasoningDecisionModel.load("altslate/certo-r1-qwen3-4b", device="cuda")
r = m.decide(
    state="A parcel weighs 2.4 kg. Shipping is $3/kg, any part of a kg rounds up. Charged $9.",
    options=[{"id":"yes", "description":"charged correctly"},
             {"id":"no",  "description":"charged incorrectly"}],
    instructions="Was the customer charged correctly?")
r["top"]        # 'yes'
r["rationale"]  # 'Round 2.4->3 kg; correct = 3x3 = $9; paid $9 -> correct. FINAL ANSWER: A'

JevBench 1.4.2 (public)

easy 0.98 · standard 0.90 · hard 0.60 — reasons far above an encoder on the hard tier (it emits an answer, trading calibration for intelligence).

~15× cheaper

concise verified reasoning: ≈32 vs ≈489 generated tokens per decision, ~8–11× lower routed latency, at non-inferior accuracy.

Same API

ReasoningDecisionModel.load().decide(state, options) — drop-in with the calibrated encoder; escalate on low confidence.

one model

One model, three decisions, calibrated probabilities.

The unified decider: one Qwen3-4B backbone, one scoring head — Choice, Score and Noul, probabilities-first.

A single head answers all three decision shapes — pick one of many (Choice), rate on a scale (Score), or yes/no (Noul) — and always returns a probability distribution, not just a label. It can answer in one fast forward pass, or write a short rationale first; both paths return a calibrated distribution, so reasoning improves the answer without giving up trustworthy confidence. Software reads the distribution into the typed response you asked for.

Strong where it counts

On the decision types it’s trained for: Choice acc 0.93–0.97 (incl. unseen domains), Noul 0.88–0.92, Score up to 1.00 — and it holds this on new domains and wordings.

Confidence you can trust

Probabilities-first, calibrated by construction: ECE 0.008–0.05 on matched data. The number means what it says — so you can abstain, escalate, or trade quality against cost.

Fast, or reasoned — one API

A cheap prompt-token read for the easy majority; a brief rationale when it helps. Both emit a distribution. decide(state, options, kind, mode).

Where it shines: train it on the decision types you’ll actually serve — routing, triage, intent, eligibility, ordinal scoring, verification — and it is accurate and well-calibrated across all three shapes from one endpoint. It’s built for the decisions you can define, not for zero-shot transfer to arbitrary unseen specifications; give it examples of a decision type and it does that type well.

We checked this on a different source (tool-calling + science decisions it never trained on): picking the right action from described options transferred almost perfectly zero-shot (0.99), while a brand-new decision type — “call a tool now, or ask first?” — needed a few examples before it became reliable and calibrated (0.57 → 0.88). Choice over described options generalizes; new rubrics and new decision types are learned, not assumed.

honest scope

What it’s for — and what it isn’t.

Good fit

  • Narrow decisions you can define or simulate: routing, triage, intent, verification, ordinal scoring.
  • Cases where trustworthy confidence matters — abstention, escalation, cost-aware choice.
  • Runs locally on a small encoder; no large model, no API.

Not this

  • A general, world-knowledge decision model — a small encoder is calibrated, not omniscient.
  • Open-ended generation — certo returns typed probabilities, not free text.
  • A drop-in for tasks you can’t generate or collect grounded data for.