Inspiration
Most coding tools explain what went wrong in a program. Fewer investigate why the learner thought the program was correct.
ProofLadder was inspired by that gap—and by research showing that learners can produce a correct answer while still holding an incorrect mental model. We wanted to build a tool that treats debugging as an investigation into reasoning, not just a search for syntax errors.
What it does
ProofLadder takes a learner's code and original problem statement, then follows:
$$ \text{Theory} \rightarrow \text{Experiments} \rightarrow \text{Verdict} \rightarrow \text{Drill} $$
GPT-5.6 proposes a falsifiable hypothesis about the learner's reasoning. An adversarial verifier designs experiments to challenge that hypothesis. The learner's program and a hidden reference are executed in a constrained sandbox, and the evidence is recorded in an append-only ledger.
A diagnosis is confirmed only when the experiments support it. Otherwise, ProofLadder explicitly marks the result as refuted or unverified instead of guessing.
After a confirmed diagnosis, the system generates a small, validated practice problem targeting the same reasoning gap. Learners can also explore visual traces, review refuted theories, download an evidence dossier, and track progress through a skill ladder.
ProofLadder is not a chat tutor and does not reveal a corrected solution to the original problem. Its goal is to help learners repair the mental model behind the bug.
How we built it
The application uses a zero-build browser interface connected to a local Python standard-library HTTP server.
The pipeline combines:
- GPT-5.6 through the OpenAI Responses API
- Strict JSON contracts for diagnoses, experiments, drills, and ledger events
- Python AST fingerprinting to identify structural signals in code
- A constrained local sandbox with timeouts and output limits
- Hidden reference execution to ground expected behavior
- Append-only JSONL evidence ledgers
- A persistent skill ladder organized by reasoning gap
- A browser UI for live investigations, visual traces, evidence review, and dossier export
The system is intentionally fail-closed. If the model, network, or credentials are unavailable, ProofLadder does not substitute a canned diagnosis or pretend that a result was verified.
Challenges we ran into
The hardest challenge was preventing confident language-model guesses from becoming educational "facts." A plausible explanation is not the same as a proven explanation, so every theory had to face adversarial experiments.
We also had to ensure that the verifier's reference solution could ground the experiments without leaking into the learner-facing interface. The sandbox needed to execute both learner and reference programs safely while keeping outputs bounded.
Finally, we designed the demo to remain honest when GPT-5.6 is unavailable. An explicit unverified result is more useful—and more trustworthy—than fabricated evidence.
Accomplishments that we're proud of
We built a complete evidence-driven learning loop instead of a prose-only explanation tool.
On a measured eight-case live corpus:
- Final diagnosis accuracy was \( 7/8 \).
- Verified precision was \( 5/6 \).
- Two cases became honest abstentions rather than unsupported confirmations.
The system recorded its one high-confidence miss instead of hiding it.
We are especially proud of the evidence ledger, refuted-theory graveyard, visual learning lab, validated drills, and downloadable investigation dossier. These features make the reasoning process inspectable for learners, mentors, and judges.
What we learned
The biggest lesson was that confidence is not evidence. In fact, the most dangerous failure was a diagnosis that was wrong while being highly confident.
We also learned that abstention is a product feature. Saying "the evidence is insufficient" protects the learner from practicing the wrong concept.
Structured contracts were equally important. They made the boundaries between model reasoning, code execution, validation, and UI presentation explicit—and made failures easier to detect.
Most importantly, the next educational step should be tied to the learner's reasoning gap, not just the error message they saw.
What's next for ProofLadder
Next, we want to expand the misconception corpus, improve calibration, and capture more refute-then-revise investigations.
We also plan to extend the evidence loop beyond Python through pluggable language runners, beginning with JavaScript and Java. The current development branch explores applying the same approach to mathematics, physics, chemistry, biology, English, and data science.
Longer term, ProofLadder could support authenticated classroom deployments, container-isolated execution, LMS integrations, and teacher-facing misconception analytics.
The metric we ultimately care about is not just whether a learner gets one answer right—it is whether they stop repeating the same reasoning mistake.
Log in or sign up for Devpost to join the conversation.