WEV

Jevify your local model. See it, calibrate it, trust it. For free.

Comment

Inspiration

Every laptop can now run a real model. SmolLM2-135M lives in a browser tab. Its brothers fit on a phone. But nobody ships them for anything that matters. Because raw local models only do one thing: generate text. Confident text, even when they're guessing. There's no I'm not sure. No typed output. No line where your app is allowed to act and where it must stop.

We stopped trying to make the small model smarter and built the layer it was missing: a jevifier.

To jevify a model means: give it a typed decision mode, look at its actual probabilities, calibrate a gate on real runs, and export that gate as plain code. SmolLM2 and its brothers are just our first test subjects: the proof that one method can jevify a whole family.

What it does

WEV jevifies any local model. For free. See it, type it, calibrate it, gate it. Drop in a model, get back a shippable decision:

1. SEE - /live Type a prompt. WEV runs the model on-device and opens it up: next-token probabilities, full-vocab entropy in bits, margin between #1 and #2. Not the string — the doubt underneath it.

2. TYPE - decide() Raw generation becomes a typed decision. A Choice picks one label (refund / review / reject), a Score rates ordered levels, a Noul answers yes/no. Full-sequence log-probs per label, softmaxed over the closed set. Same interface for browser transformers.js, a local server, or a file — so live and replay can never diverge.

3. CALIBRATE + GATE - lib/kernel.ts pnpm capture runs your model on ~100 labelled items. pnpm calibrate picks thresholds on half A, reports on held-out half B. The gate is one readable rule: AUTO iff confidence >= X AND entropy <= Y, else ESCALATE. Malformed output fails closed. No randomness. No network. No vibes.

4. SHIP - export + /verify Copy Gate = self-contained function with your thresholds + real test cases. Copy Receipt = sha256 record of every decision. Paste anywhere, re-run, re-check. Flip one char and it reads TAMPERED.

Demo task is 40 support tickets. The product is the jevifier around them.

How we built it

We built WEV backwards: kernel first, UI last.

The kernel (lib/kernel.ts, ~600 lines, zero deps) enforces 5 invariants: distribution must sum to ~1 and be sorted, entropy is exact Shannon bits with no pre-rounding, close races never promote to confident, flat tops < 0.30 always abstain, identical input = identical digest. Then evaluateTriageGate() applies the calibrated thresholds.

The decider (lib/decider.ts) is what makes it model-agnostic. It only reads probabilities + entropyBits. It doesn't care if they came from SmolLM2-135M in the browser, 360M in Node via onnxruntime-node, or a CSV. One verdict path for all adapters.

Then we jevified two brothers to prove it:

  • Little brother 135M: 33/100 alone. Calibrated to conf >= 0.5, entropy <= 6 bits → held-out 5/50 AUTO at 10% coverage, 60% accuracy vs 40% always-trust, 2 wrong autos.
  • Big brother 360M: 34/100 alone. Calibrated to conf >= 0.5, entropy <= 3 bits → held-out 14/50 AUTO at 28% coverage, 50% vs 32% baseline, 7 wrong autos.

Different entropy scale, different card, same code. Thresholds live per model ID and are never reused. pnpm claim:verify replays all 116 fixtures and re-derives every number as we predicted.

Challenges we ran into

Making it about the jevifier, not the tickets. Early versions looked like a support app. We had to rip out everything that wasn't inspect → type → calibrate → gate → export, and say plainly: tickets are synthetic test data, the gate is the product.

Confidence lied, per model, differently. The 135M was 88% confident and wrong. Entropy caught it. Then the 360M showed up with a totally different entropy range — our 135M thresholds (≤6 bits) would have auto-handled garbage. That bug became our core rule: the method travels, the numbers don't.

Browsers hate 130MB. CDN runtime + ONNX weights + OOMs + Node-vs-browser divergence. Fix was brutal simplification: three adapters, one DeciderOutput type, deterministic replay for demos.

Failing closed is 90% of the work. Unsorted probs, sum 0.97, NaN entropy, coin-flip flats - all wanted to render as answers. We now have hand-written adversarial fixtures that must be REJECTED, plus 5 invariants checked in UI and in verify:evidence. 116/116 green.

Accomplishments that we're proud of

  • Jevified a family, not a checkpoint. Two SmolLM2 brothers, two gate cards, one pipeline. Nobody can say we tuned to one model.
  • A gate that leaves WEV. Copy Gate pastes into a blank file and passes 3/3 evidence tests.
  • Honest numbers on screen. 5 auto, 45 escalated. Wrong-auto gallery with ticket text + confidence. Limits panel that says: synthetic, 50 tickets, one task. We show misses because a gate you can audit is the whole point.
  • Receipts that bite. sha256 + verdict recompute + 1-byte tamper detection on /verify. Support leads don't trust accuracy claims. They trust receipts.
  • Fully reproducible. No key, no server: pnpm capture && pnpm calibrate && pnpm claim:verify. 116/116 PASS. Live at wev-one.vercel.app, zero signup.

What we learned

  • Small models don't need more prompting. They need a smaller job: a closed label set, scored from real log-probs, with an explicit escalate path.

  • Text hides uncertainty; distributions show it. Once you've seen a 47% top-prob with 5.8 bits of entropy, you stop trusting fluent strings forever.

  • Calibration has to be held-out and per-model. Anything else is self-deception. Seeded 50/50 felt pedantic until it saved us from shipping 135M thresholds on the 360M.

  • And trust comes from what you admit.

What's next for WEV

WEV stays a jevifier. Tickets were test one.

More brothers: Qwen2.5-0.5B, Llama-3.2-1B, Gemma-2-2B: each gets its own card from the same capture → calibrate loop. Custom HF model ID box and local folder drop (config + tokenizer + ONNX, nothing uploads) graduate from Experimental to supported.

One command: npx wev init — drops gate.ts + thresholds.json + gate-card.json into any app. The export already works; we're turning it into an SDK.

Real data: Partner with one support team on ~500 de-identified tickets. Replace our synthetic 100, recalibrate live in /calibrate, publish what a tiny local model can honestly auto-handle on real traffic.

We don't make local models smarter. We make them shippable.

Built With

Share this project:

Updates

Submission history