ooo Squirrel

An executive-function debugger for small language models

Comment

SQUIRREL - Project Story

An executive-function debugger for small language models.

Inspiration

Small language models fail in a strange way. Very often they already know enough to solve a problem, yet they still get it wrong - because one weak assumption slips into the reasoning early, survives unchallenged, and quietly corrupts everything that follows. That does not look like missing knowledge. It looks like a failure of executive function - the mind's capacity to challenge, prune, and resolve its own thoughts before acting on them. (Our internal codename was Project Methylphenidate, after the medication that supports exactly that faculty.)

So we asked a falsifiable question: if the failure is executive rather than factual, can an external executive-function layer, applied only at inference time, recover the capability - with no fine-tuning, no retrieval, no change to the weights, and no larger model writing the answer?

The classic bat-and-ball problem is the failure in miniature. "A bat and a ball cost \$1.10; the bat costs \$1.00 more than the ball; how much is the ball?" The tempting answer is \$0.10. It is wrong, and it is wrong because of one unchecked assumption. The constraint says:

$$x + (x + 1.00) = 1.10 \;\Rightarrow\; 2x = 0.10 \;\Rightarrow\; x = 0.05$$

A model that "knows" algebra still answers \$0.10 when nothing forces it to test the assumption. SQUIRREL is the thing that forces the test - and lets you watch it happen.

What it does

SQUIRREL runs the same task through the same model under three cognitive-control conditions and shows them side by side:

  • Naked - the unmodified model answers in one pass. The assumption flows straight into the answer, unchallenged.
  • Guided (executive) - every intermediate claim is classified as Fact, Assumption, Possibility, or Contradiction. Unsupported claims are challenged, contradictions are refuted at an independent audit, and refuted claims are kept out of the grounded answer.
  • Squirrel (divergent) - deliberately explores unusual associations, then returns to a convergence checkpoint so creativity does not collapse into incoherence.

Every claim carries content-addressed provenance; every event is hash-chained; and one honest control metric, contamination count, reports how many sandboxed or refuted claims leaked into the final answer. The scaffold's job is to keep that at $0$. A GPT-5.6 diagnosis panel then reads the naked and scaffolded traces and explains, in plain language, where the reasoning first diverged and what the executive layer did about it.

How we built it, with Codex and GPT-5.6

The most unusual part of this project is that Codex did not build a wrapper around a finished idea - it helped originate the idea. Codex turned the executive-function analogy into a falsifiable experiment: it designed the test conditions and controls, built the content-addressed claim/provenance engine and the model-qualification harness, and ran the experiments that produced our results.

GPT-5.6 is the laboratory instrument, never the answer engine. The subject (small) model always produces the answer; GPT-5.6 generates adversarial task variations and diagnoses reasoning trajectories - it analyses the experiment, it does not sit the exam. Keeping that boundary strict is what makes the result honest.

The Build Week extension - the trace normalizer, the live scaffold-on/off runner, the GPT-5.6 diagnostic, and a runnable three-column product experience - was built by Codex in a dedicated session (cited via /feedback), isolated in a buildweek/ layer that never touches the frozen research corpus.

We are transparent about the full toolchain. When the Codex usage cap was reached mid-experiment, Claude (via Claude Code) continued portions of the implementation and wrote the build specs, and a second Claude model session served as an independent verification gate - a "worker" that checked every artifact byte-for-byte before it was accepted. Builder, gate, and human operator were kept as separate roles with separate authority. That separation of duties is not a workaround we are hiding; it is a genuine multi-agent workflow, and it is part of why we trust the provenance.

What we learned

  • The honest claim is control, not accuracy. SQUIRREL reliably makes a model's reasoning inspectable and controllable - you can watch the bad assumption get caught and quarantined. We were deliberately rigorous about what we did not claim: our core-mechanics run was mocked, and we labelled it "synthetic engineering evidence, not empirical proof" of an accuracy effect. Any single-task recovery in the demo is illustrative, on that task only.
  • Provenance was harder than the model. The real engineering was discipline: content-addressing every claim with a canonical sha256, hash-chaining every event, deriving the four-way label from claim kind and verification state, and refusing - structurally - to let an unverified claim reach the answer.
  • Multi-agent separation of duties pays off. A builder that also grades its own work will pass itself. An independent gate caught real discrepancies (a claimed-complete table that was empty; CRLF claimed as LF) at the byte level.

Challenges we faced

  • The Codex cap forced the multi-agent split mid-flight; we turned that constraint into a cleaner workflow (Codex builds the cited core; specs and non-core plumbing done around it).
  • Scientific integrity vs. demo speed. The research corpus runs under strict one-shot, frozen-artifact discipline. We protected it by building the entire Build Week demo as a separate, read-only replay layer plus an isolated live runner - the frozen experiment is never edited to make a nicer demo.
  • Resisting the overclaim. The easy pitch was "small model gets smarter." The honest, and ultimately stronger, pitch for a developer tool is: see and control what your model is actually assuming.

Built With

  • ajv
  • claude
  • claude-code
  • codex
  • css
  • es-modules
  • gpt-5.6
  • gpt-oss-20b
  • hash-chained-event-log
  • html
  • javascript
  • json-schema
  • multi-agent-workflow
  • node.js
  • node:test
  • ollama
  • openai-api
  • playwright
  • sha-256-content-addressing
  • smollm3
  • uuidv5
Share this project:

Updates