CounterLab: The Scientific Debugger for Beliefs

Ask like chat. Prove it like science.

Inspiration

AI tutors can give excellent explanations, but a learner can still walk away with the same mistaken mental model.

That problem becomes especially serious in machine learning. A notebook may report an impressive metric while evaluating the wrong thing. For example, a random row split can place records from the same customer in both the training and test sets. The reported score may be mathematically correct, yet it does not answer the question the learner actually cares about:

Will this model work for a completely new customer?

I did not want to build another chatbot that simply explains leakage, or another notebook linter that points at suspicious code. I wanted learners to make their belief testable, commit to what they expect, and confront controlled evidence produced from their own work.

Chatbots explain. CounterLab lets reality answer.

What it does

CounterLab turns a learner’s question, claim, or supported Jupyter notebook into a verified learning experiment.

The experience follows six stages:

Question The learner states what they think a result proves. CounterLab connects that claim to exact notebook cells, outputs, metrics, and schema evidence.
Prediction CounterLab presents two competing explanations. The learner confirms or edits the explanation that matches their current thinking, then locks a prediction before any experimental result is revealed.
Test Codex compiles a bounded experiment designed to separate the competing explanations. A fixed scorer and independent verifier check that the test is fair, controlled, reproducible, and capable of answering the learner’s actual question.
Boundary The learner explores where the observed pattern changes or stops applying. Instead of replacing one oversimplified rule with another, CounterLab helps the learner discover the conditions under which the rule holds.
Apply The learner receives a meaningfully different transfer case. A fixed evaluator checks whether the learner can apply the revised principle without relying on the original notebook or wording.
Repair Only after transfer succeeds does CounterLab unlock a minimal notebook correction. The original notebook remains untouched, and the repair is applied to a separately verified copy.

CounterLab currently supports two machine-learning misconception families:

Entity leakage, where familiar entities appear in both training and test data.
Class imbalance, where high accuracy can hide failure on rare but important cases.

Every experiment ends with one of three Evidence Verdicts:

SUPPORTS
INCONCLUSIVE
REJECTED

A rejected experiment releases no authoritative result. An inconclusive experiment is shown honestly rather than being forced into a success or failure narrative.

At the end of a completed session, the learner receives a visual Reasoning Diff showing:

what they originally believed;
what they predicted;
what the experiment measured;
where the rule changes;
whether the idea transferred;
what changed in the repaired notebook.

The learner can also export a Proof Capsule containing the prediction, experiment specification, signed result, verifier reports, Boundary Map, transfer outcome, repair history, versions, hashes, and replay information.

How I built it

CounterLab is a solo project.

I built the complete system myself with Codex as my primary engineering collaborator. The repository now contains more than 550 commits. Reaching that level of implementation, testing, documentation, debugging, and iteration within the hackathon window would not have been realistic for me without Codex.

Codex helped me move quickly across:

architecture;
React and TypeScript implementation;
Python services;
schema design;
Cloudflare deployment infrastructure;
testing;
accessibility;
mutation evaluation;
browser automation;
release documentation;
repeated debugging and refinement.

Codex made the volume of work possible, but I retained control over the product direction, authority boundaries, learning design, supported scope, verification model, safety rules, and final engineering decisions.

CounterLab deliberately separates generation from authority.

GPT-5.6

GPT-5.6 is responsible for semantic and educational reasoning. It can:

analyze sanitized notebook evidence;
connect claims to exact artifact references;
propose competing explanations;
ask bounded clarifying questions;
shape learner-facing explanations;
help compose an approved visual learning scene.

GPT-5.6 cannot:

invent verified values;
execute uploaded notebooks;
decide whether an experiment is valid;
grade transfer;
approve a repair;
declare mastery.

Codex

Runtime Codex compiles candidate experiments into a strict, allowlisted Experiment IR. It can also produce a separate artifact-specific Patch Plan after the learner passes transfer.

Codex may propose or repair:

the controlled experiment plan;
the declared intervention;
approved visual composition;
a public rationale;
an artifact-specific notebook repair plan.

Codex cannot:

define metric formulas;
modify the fixed kernels;
access hidden verifier logic;
grade its own output;
approve its own repair;
install arbitrary packages;
execute unrestricted code in the hosted path.

Fixed experiment scorer

A deterministic scorer evaluates candidate experiments before execution. It rejects plans that:

change several uncontrolled variables;
fail to match the deployment question;
omit necessary observables;
cannot meaningfully separate the competing explanations;
exceed the supported Subject Pack.

Fixed numerical kernels

Python, NumPy, pandas, SciPy, and scikit-learn calculate every authoritative result, including:

train and test splits;
entity overlap;
accuracy;
precision;
recall;
F1;
ROC-AUC;
PR-AUC;
confusion matrices;
threshold sweeps;
prevalence sweeps;
Boundary Map cells;
transfer scores;
repaired notebook results.

The language models never provide the numerical values shown as verified evidence.

Frozen verifier

A separate verifier checks:

technical validity;
experimental validity;
epistemic validity;
controlled comparisons;
result-to-visual bindings;
reproducibility;
resource limits;
patch scope;
provenance consistency.

If the verifier finds that a proposed experiment cannot support the intended conclusion, CounterLab blocks the result and sends a structured counterexample back to Codex for repair.

Learner authority

The learner controls:

the original claim;
which explanation reflects their thinking;
the prediction;
confidence;
interpretation;
revised rule;
transfer response;
repair approval.

CounterLab does not reduce the learner to approving an AI-generated answer.

Technical architecture

The learner application uses:

React;
Vite;
strict TypeScript;
accessible, typed UI components;
responsive layouts;
keyboard navigation;
reduced-motion support.

Zod and JSON Schema enforce fail-closed contracts between every major layer.

A Cloudflare Worker coordinates the application. It uses:

D1 for sessions, state, job records, and append-only evidence events;
R2 for private notebook artifacts, repaired copies, and proof exports;
Durable Objects for bounded Container-job coordination;
reconnectable event streaming for long-running compilation and verification work.

The bounded runner integrates with Codex App Server through structured stdio JSONL events. Public activity displays concise stages, file changes, verifier findings, and repair outcomes without exposing private reasoning.

CounterLab’s generative UI uses an allowlisted LabSceneV2 contract. A generated scene may choose approved visual components and bind them to verified result paths. It cannot provide the result data itself.

If a scene, result hash, operation, or binding disagrees with the signed evidence, CounterLab withholds the visual.

Challenges I ran into

Preventing generated systems from validating themselves

The hardest design problem was circular authority.

A model can produce an experiment, a test, an explanation, and a confident verdict that all share the same mistaken assumption. That may look convincing while proving nothing.

I therefore separated:

proposal;
experiment selection;
numerical computation;
verification;
learner judgment.

Models propose. Fixed systems decide what becomes authoritative.

Preventing prediction leakage

Prediction only matters when it is genuinely recorded before the reveal.

CounterLab therefore prevents verified values and Boundary Maps from becoming available until the learner’s prediction has been immutably committed. This restriction remains in place across:

refresh;
back navigation;
reconnection;
session recovery;
replay.

Keeping Sample, Replay, and Live modes honest

A deterministic sample is useful, but it must never look like a fresh model result.

A recorded replay is useful, but it must remain read-only and visibly labelled.

A live session must fail clearly when the runner or model is unavailable. It must never silently replace a failed live result with sample evidence.

CounterLab keeps these modes separate at the contract, storage, API, and interface levels.

Safely handling notebooks

Uploaded notebooks are untrusted data.

CounterLab never executes uploaded cells during intake. It accepts only a documented subset, removes or refuses active content, validates file type and size, extracts evidence references, and returns typed reasons when an artifact is unsupported.

Producing reproducible proof across several runtimes

CounterLab crosses:

the browser;
React and TypeScript;
the Cloudflare Worker;
D1;
R2;
Containers;
Codex;
Python;
scientific libraries.

Keeping evidence consistent across these boundaries required:

canonical serialization;
content hashes;
append-only event chains;
fixed seeds;
versioned contracts;
mutation tests;
idempotent callbacks;
source-bound runner images;
deterministic Proof Capsules.

Operational safety

The most painful challenge was operational safety during development.

A Codex Ultra session crossed its intended filesystem scope and deleted files across my home directory. My desktop environment became unusable, and I lost roughly twelve hours restoring the operating system, development tools, and surviving project state.

That incident changed how I approached CounterLab.

Agent safety could no longer be a prompt-level promise. I introduced stronger controls around:

repository-only filesystem access;
path-containment checks;
non-root execution;
bounded resources;
isolated tool state;
explicit destructive-action approval;
external checkpoints;
fail-closed release gates;
immutable evidence;
limited runtime authority.

The same lesson shaped the product itself. GPT-5.6 and Codex can reason, investigate approved evidence, and propose work. They cannot freely modify authoritative systems or decide what becomes trusted evidence.

These guardrails were not added as theoretical architecture. They were added after I experienced the real cost of an agent crossing its authority boundary.

Accomplishments I am proud of

I am proud that CounterLab is more than an AI explanation wrapped around a chart.

I built:

a complete Question, Prediction, Test, Boundary, Apply, and Repair loop;
immutable prediction-before-reveal behavior;
evidence-linked competing mental models;
deterministic experiment selection;
fixed numerical authority;
independent technical and epistemic verification;
interactive, accessible Boundary Maps;
transfer tasks that must pass before repair unlocks;
minimal notebook repairs applied only to verified copies;
evidence-bound generative UI;
replayable Proof Capsules;
learner-facing Reasoning Diffs;
explicit Sample, Live, and Replay authority labels;
responsive learner and Judge experiences;
keyboard and reduced-motion support;
two complete machine-learning Subject Packs;
held-out notebook evaluation;
mutation testing;
reconnectable hosted execution;
source and version provenance throughout the result chain.

I am also proud that CounterLab can say:

The evidence is inconclusive.

It can refuse to release a result when the experiment is weak, confounded, unsupported, or unable to distinguish the competing explanations.

That restraint is one of the project’s most important features.

What I learned

I learned that making an AI system more agentic should not mean giving it unlimited authority.

GPT-5.6 becomes more useful when it can reason over approved evidence and shape a learning experience. Codex becomes more useful when it can compile and repair bounded technical artifacts. Neither should become the numerical oracle or final verifier.

I also learned that prediction changes the emotional quality of learning.

A result matters more when the learner has already committed to what they expect. The difference between prediction and observation creates a reason to reconsider the original mental model.

I learned that a scientific-looking interface is not automatically scientific. Every metric, chart, animation, comparison, and Boundary cell must resolve to the exact verified payload that authorized it.

Most importantly, I learned that correcting an answer is not the same as changing understanding.

A useful educational system should help the learner:

identify the assumption;
make it falsifiable;
observe controlled evidence;
discover the rule’s limits;
apply the revised model somewhere new.

What is next for CounterLab

My immediate priority is release qualification.

Before expanding the project, I want to complete:

exact-source public deployment validation;
full desktop and mobile browser journeys;
final reconnect and download testing;
release checks in the locked environment;
a small, consented learner study.

I currently make no measured learning-effect or mastery claim.

The learner study will focus on:

completion;
confusion points;
immediate transfer;
confidence calibration;
delayed retention where feasible.

After the existing notebook paths are fully reliable, I plan to add one cross-domain proof: a guided free-fall Subject Pack.

That pack would use:

fixed numerical integration;
an analytic reference solution;
dimensional validation;
signed motion data;
drag and vacuum conditions;
an interactive Boundary Map;
deterministic transfer.

I also want to make the bounded generative UI more expressive, so different misconceptions can receive different visual explanations without letting presentation become evidence authority.

Only after the scientific core is stable will I add shareable, no-account transfer challenges.

CounterLab’s long-term goal is simple: