calibrated decision models
certo.
Small models that know how sure they are.

A toolkit for building narrow, calibrated decision models — trained on synthetic data with a known answer key, so the confidence they report is trustworthy by construction, not patched on afterward.

pip install certo research preview MIT non-generative · typed probabilities

Read the technical report →

Inspired by Jev / “System-1” decision models. Independent project — not affiliated with TypeSafe.

the problem

The answer is easy. The confidence is the hard part.

A decision model earns its keep when it says “route to billing, 0.82” and the 0.82 is real — so you can escalate on doubt or trade quality against cost. Train on hard labels (the usual way) and the model still picks well but turns overconfident; you’re left patching calibration after the fact.

Train on the distribution

certo trains against the full answer distribution (a proper scoring rule), so probabilities mean what they say — calibrated by construction.

Grade against the truth

Because the data is generated from a known posterior, every model is scored against the exact answer — not just accuracy, but fidelity.

how it works

A known-answer world, dressed in language.

You define a decision as a synthetic world whose answer is computable. certo renders it to natural language, and a small encoder learns to read the state and each option and emit calibrated, typed probabilities. The answer key both trains and grades.

01
Synthetic world
a decision with a known, closed-form answer
02
Rendered to text
state + runtime options, many wordings
03
Small encoder
ModernBERT reads state & options
04
Typed probabilities
choice · binary · score · multi-label
stateEnglish encodercalibrated p exact answer key — trains & grades (the model never sees it)
The model only ever reads the text; the exact answer is used to build the training target and to grade fidelity.
evidence — controlled worlds

Calibration recovered from plain English.

Measured against the exact posterior (KL, lower is better) on held-out wording. Training on the full distribution lands essentially on the answer; single-label targets sit near “ignore the evidence.”

0.004
KL to exact answer
(soft targets, from text)
≈ Bayes
accuracy at the optimal ceiling
0.00
option-order sensitivity
(invariant by construction)
0 0.8 KL to exact answer ignore-evidence baseline 0.605 0.004 soft 0.72 sampled 0.61 argmax
ModernBERT reading English · held-out wording · robust across seeds. And in a routing task, only calibrated probabilities support cost-aware choice — accuracy alone can’t.
quickstart

Define a decision, generate data, ship a classifier.

The interface certo ships (research preview — API stabilizing).

# 1 — describe your decision as a known-answer synthetic world
from certo.synth import EvidenceWorld
world = EvidenceWorld(classes=["billing", "tech", "refunds"], features=12)
data  = world.generate(n=20_000)          # (state, options, EXACT target)

# 2 — train, calibrated by construction (soft/distributional targets)
from certo import train
model = train(data, backbone="ModernBERT-base", target="soft")
model.save("my-router")

# 3 — use it locally: state + runtime options -> calibrated probabilities
from certo import DecisionModel
m = DecisionModel.load("my-router")
r = m.decide(state="card charged twice, refund please",
             options=[{"name":"billing", "description":"charges & payments"},
                      {"name":"tech",    "description":"app problems"}],
             kind="choice")
r.probs        # {"billing": 0.86, "tech": 0.08, "OUT": 0.06}  — calibrated
r.abstain      # True when the top probability is below your threshold
honest scope

What it’s for — and what it isn’t.

Good fit

  • Narrow decisions you can define or simulate: routing, triage, intent, verification, ordinal scoring.
  • Cases where trustworthy confidence matters — abstention, escalation, cost-aware choice.
  • Runs locally on a small encoder; no large model, no API.

Not this

  • A general, world-knowledge decision model — a small encoder is calibrated, not omniscient.
  • Open-ended generation — certo returns typed probabilities, not free text.
  • A drop-in for tasks you can’t generate or collect grounded data for.