Inspiration
I kept coming back to one uncomfortable question:
Why should a narrow win at one token decide an entire solution?
A small language model can be genuinely uncertain at the exact moment two reasoning plans diverge. Greedy decoding keeps the slightly more likely token and permanently deletes the other path. The model may already contain a better solution; we simply never let it finish that thought.
That became the idea behind Search Before Trust:
When a model hesitates, search the plausible paths first. Trust one only after the solutions are complete.
What it does
Search Before Trust, or SBT, is a deterministic, training-free decoding framework for small local language models. On the frozen GSM8K test slice it takes Qwen3-0.6B from 43.4% to 52.2% without any fine-tuning, reward model, or larger model in the loop.
At every generated position, SBT looks for a plausible alternative satisfying:
$$p_{\text{alternative}} \geq \alpha\, p_{\text{chosen}}$$
Instead of expanding every possibility, it schedules only the strongest alternatives inside a globally bounded search tree.
The decoding process is:
- Generate the ordinary greedy solution.
- Record close token alternatives where the model hesitates.
- Expand the strongest alternatives across a limited number of search waves.
- Let the same small model complete every retained path.
- Deduplicate equivalent candidate programs.
- Execute each unique candidate and reject programs that do not run.
- Rank the eligible candidates using the model's average log-probability over the complete solution.
The original greedy answer always remains candidate zero.
SBT does not fine-tune the model, train a reward model, expose the gold answer during inference, or ask a larger model to select the result. The candidate search and final choice use only the small model's probabilities, deterministic search logic, and a task-local executability check.
An SBT result contains more than an answer. It shows which paths were considered, where they forked, what each program returned, why candidates were rejected, how the winner ranked, and how much search was performed.
How we built it
I built SBT as a developer tool, not just a benchmark script. The repository includes a typed Python API and CLI, a local Hugging Face backend, reusable backend and verifier protocols, deterministic branch scheduling and replay, structured fallback and no_answer states, machine-readable candidate and execution and ranking and budget traces, exact-budget evaluation utilities with paired statistics, and frozen evidence receipts with an offline release verifier.
Before the main evaluation, I froze the model revision, dataset revision, search configuration, evaluator, and inference boundary. The final inference function received only an item ID and the problem text; reference answers were joined later for offline scoring. That freeze is what makes the result a measurement rather than a search over configurations.
I used Codex as the primary engineering environment, with GPT-5.6 Sol as the main implementation and scientific-review model. I steered the product direction, experimental scope, freeze decisions, and final release boundaries. The workflow built and refined the typed solver and branch scheduler, the Hugging Face and deterministic scripted backends, verifier and executor boundaries, trace schemas and CLI and Python integration, exact-budget baseline allocation, paired statistics and decontamination checks, selector and cache audits, frozen-result receipts, and the documentation and tests around all of it.
The most valuable contributions were often objections rather than features. The Codex workflow caught a failed equal-token claim, challenged a fragile selector change, retained the development result where a baseline beat SBT, and preserved the cache benchmark that rejected a speedup claim.
Primary Codex /feedback Session ID: 019f75d5-d29d-7212-8ac1-99e9c759bf47
The causal boundary is important: GPT-5.6 Sol helped build and audit SBT, but it never proposed, generated, executed, verified, scored, ranked, or selected any reported benchmark candidate. The benchmark path remained small-model-only.
Challenges we ran into
I expected tree search to be the hard part. It was difficult, but the harder problem was deciding what evidence I was willing to believe.
My first "fair" comparison was wrong
The first equal-budget experiment matched candidate counts but failed exact token equality on every item in its three-item smoke run.
It would have been easy to keep the attractive label and move on. Instead, I preserved the failed receipt with claim_equal_tokens_supported=false, rebuilt the allocator, and required exact per-item candidate and output-token checks before scaling.
Even the corrected comparison has a boundary: equal candidate counts and output tokens do not prove equal FLOPs, latency, prompt processing, executor work, memory, energy, or cost. The repository says that directly.
More search did not automatically mean better answers
On development data, wider trees found additional correct candidates while selected accuracy became worse.
That was a humbling result. The tree was doing its job; the ranker was not. It forced me to stop treating search coverage as accuracy and to audit generation and selection separately.
The least enjoyable development result was also one of the most useful: conservative self-consistency scored 51/100 while SBT scored 47/100. I kept it. It narrowed the project's claim before the held-out evaluation instead of letting me move the goalposts afterward.
The optimization that looked like a win was not a win
I implemented guarded cross-branch KV-cache reuse and measured a 90.97% reduction in physical branch-prefill token positions.
That looked like the speed story I wanted — until synchronized wall time came back roughly 1.4% slower, and small float16 differences changed parts of the branch structure.
A counter going down is not the same thing as a system getting faster.
The cache implementation remains an inspectable experiment, but it is disabled in the frozen result. SBT makes no realized speed, FLOP, energy, or cost-saving claim.
Executable does not mean correct
A candidate program can run perfectly and still solve the problem incorrectly.
The executor therefore acts only as a cheap validity filter. It blocks obvious unsafe syntax, runs trusted benchmark generations in a child process, and enforces a timeout, but I do not present it as a secure production sandbox or a proof of mathematical correctness.
That distinction is one reason the complete candidate trace matters.
Accomplishments that we're proud of
On the predeclared first 500 GSM8K test items using Qwen3-0.6B:
| Method | Correct | Accuracy |
|---|---|---|
| Search Before Trust | 261/500 | 52.2% |
| Greedy decoding | 217/500 | 43.4% |
| Conservative self-consistency | 248/500 | 49.6% |
Against greedy decoding, SBT gained 8.8 percentage points, with 55 SBT-only successes and 11 greedy-only successes. The paired result was statistically clear with an exact McNemar $p = 3.60 \times 10^{-8}$.
The stronger comparison was much closer. SBT finished 2.6 points above conservative self-consistency, but that margin was not statistically resolved, $p = .1875$.
I am publishing that result plainly. A comparison reported only when it wins is not a useful comparison.
The number I find most interesting is actually 317.
A correct executable candidate existed somewhere inside SBT's tree on 317 of the 500 items, but the frozen selector chose correctly on 261. That leaves 56 observed ranking misses.
Those are not 56 points I am claiming. They are evidence that finding a useful path and recognizing it are two different problems — and a precise map of where the next improvement has to come from.
The confidence measurements supported the original intuition:
$$0.5301 \rightarrow 0.6209 \rightarrow 0.6977$$
Those are fork-local, whole-solution, and executable-only AUROC. At the fork, confidence was barely informative. After the model finished each thought — and especially after invalid programs were removed — the same model's confidence became substantially more useful.
I also froze the unchanged method on a filtered 100-item SVAMP slice. SBT scored 82%, greedy scored 78%, and conservative self-consistency scored 84%. None of those differences was statistically resolved, so I treat that run as narrow secondary evidence, not proof of broad generality.
Evidence that can say "no"
The part of SBT I am proudest of may be the least flashy: its offline release checker.
python3 -B -m evals.release_check
This command strict-parses both frozen benchmark receipts, verifies their pinned SHA-256 hashes, and recomputes the declared results from the raw item-level records.
It performs no inference, makes no network request, and does not rewrite the evidence. If a receipt changes or a reported number disagrees with recomputation, the release check exits with an error.
I did not want the repository to ask a judge to trust a screenshot. I wanted it to be capable of objecting when its own results were misrepresented.
The evidence machinery began as protection against fooling myself. It became one of the most reusable parts of the tool.
What we learned
The thing I keep returning to is that uncertainty has timing. Token-level confidence is good enough to tell you where to look and not good enough to tell you what to pick. The distance between 0.5301 and 0.6977 is that gap, measured.
The corollary cost me a week: a wider tree creates opportunity and nothing else. On development data I made search better and accuracy worse. Finding and recognizing are different problems, and I had been treating them as one.
The last lesson I did not expect. I built the evidence machinery — traces, hashes, frozen configurations, preserved negative results — to stop myself from fooling myself. It ended up being the most reusable part of the tool, and it shapes what the system is allowed to claim about itself.
I also learned to separate building with a powerful model from quietly using one inside the result. Codex and GPT-5.6 Sol were central engineering collaborators, while the inference path remained local, inspectable, and independent of a large-model judge.
What's next for Search Before Trust: Auditable Decoding for Small LLMs
The clearest next target is ranking. Any improved selector must be fitted only on development data and tested prospectively, rather than tuned against these held-out outcomes.
After that, I want to test additional model families and tasks with inexpensive deterministic checks, replace the benchmark executor with a real production sandbox, evaluate calibrated adaptive stopping instead of fixed search depth, measure repeated per-method latency and memory and FLOPs and hardware cost, and revisit KV-cache reuse only after demonstrating both structural stability and an actual wall-time benefit.
SBT began as a way to give a small model one careful second look.
The name started as advice for the decoder. Somewhere along the way it became advice for me.
Log in or sign up for Devpost to join the conversation.