MindDiff, git diff for mental models

Inspiration

AI now writes most student homework. Grades look fine, but understanding is quietly disappearing, and neither autograders (which only report what failed) nor AI tutors (which hand over the fix and undermine learning) can tell the difference. We wanted a tool that diagnoses why a student is wrong, not just that they are, and proves its diagnosis instead of just asserting it.

What it does

MindDiff reads a student's failing Python submission, has GPT-5.6 hypothesize the false belief that produced the bug, and designs a discriminating probe for each hypothesis: a minimal experiment whose output differs depending on whether the belief is real. Probes execute in a Pyodide sandbox, and a deterministic verdict machine confirms, rules out, or marks unverifiable. An LLM's opinion can never confirm itself; only matching executed evidence can. Students get a Socratic question pointing at the experiment, never the fix. Instructors get a class wide misconception heatmap, drilldown into exactly which students hold which belief in their own words, embedding based clustering of near duplicate misconceptions, and a one click GPT-5.6 generated 5 minute mini-lesson targeting the biggest cluster.

How we built it

The core diagnostic engine, including the Pyodide sandbox, the GPT-5.6 structured output diagnosis pipeline, and the initial student flow, was built in a single Codex CLI session on GPT-5.6 Terra. Everything else, including the full exercise set, a 40 submission synthetic classroom with planted misconceptions, the instructor dashboard, rate limiting and input validation, and the Vercel/Turso deployment, was built on top of that foundation. Stack: Next.js, TypeScript, Drizzle ORM over Turso/LibSQL, Zod validated structured outputs from the OpenAI SDK, and Pyodide for sandboxed Python execution with zero server side code execution.

Challenges we ran into

The trickiest bug wasn't in the UI, it was in the prompt. GPT-5.6 was initially filling probe predictions with what the student would guess, not what execution would actually show if the misconception were real, which silently inverted every verdict. We caught it by manually stress testing the live pipeline against real submissions, not just unit tests. We also found and fixed a cost exploit where the diagnose endpoint could be triggered on empty or fabricated failing test evidence, and added per IP and global rate limits since the public demo has no login system to attach a per user quota to.

Accomplishments that we're proud of

The verification loop works end to end on real, unscripted input, including edge cases we specifically stress tested. a genuine Python syntax error gets diagnosed rather than crashing, and an infinite loop times out gracefully with the system honestly reporting "unverifiable" rather than fabricating a confident answer when its own probe can't discriminate. That honesty guarantee, working live, is the whole thesis of the project.

What we learned

The riskiest part of an LLM verification pipeline isn't the model refusing to answer, it's the model answering confidently and wrong in a way that looks correct. Building a system that can honestly say "I can't verify this" turned out to be the hard and important part.

What's next

Real classroom piloting with instructor consent, LMS integration, and extending the misconception taxonomy beyond Python fundamentals to domains like SQL and statistics.

Built With

• api
• codex
• css
• drizzle
• gpt-5.6
• libsql
• next.js
• openai
• orm
• pyodide
• python
• react
• tailwind
• turso
• typescript
• vercel
• vitest
• zod