Misconception Map

From a graded exam to what to teach next, an AI that finds the misconception, shows its evidence, and never guesses.

Comment

Inspiration

Grading is only half the work. After the red pen comes the real question: what do I reteach tomorrow? A stack of corrected exams tells you who failed, but not why.

Research has been saying this for decades: student errors are not noise. Brown & Burton (1978) showed that wrong answers in arithmetic follow systematic procedural bugs. Sleeman (1984) distinguished those systematic misconceptions from one-off slips. A student who writes

−3(x+4)=−3x+12 −3(x+4)=−3x+12

didn't have a bad day. They apply a rule: "the minus sign belongs to the first term only" and they will apply it again next week, on a different exercise, unless someone names it and teaches against it. That rule, not the wrong answer, is the thing a teacher needs to see. So we built the tool that finds it.

What it does

Misconception Map is a teacher-facing diagnostic workspace. You upload an exam (PDF, photo, or typed) and the handwritten student copies. GPT-5.6 extracts the exam structure, transcribes each student's work step by step, diagnoses the misconception behind each error always backed by a quoted line from the copy, and helps mark the exam. The class view ranks the most frequent difficulties and names the students behind each one. From there, one click writes a "Teach This Tomorrow" brief, targeted practice sheets, or a full follow-up evaluation that retests every mistake the class actually made, in the exam's own language. The Prediction Lab goes further: it turns repeated errors into falsifiable per-student models and tests them on unseen work.

Three rules are enforced in the schema, not just the UI: the AI proposes and the teacher decides (no AI score reaches the gradebook before explicit validation, one-way PROPOSED → VALIDATED transition guarded by SQL triggers); a wrong answer alone is never a misconception (a label requires a grounded incorrect step and confidence above 0.72); and when evidence is weak, the engine visibly abstains instead of guessing.

How we built it

Codex wrote most of the code; we set the boundaries and kept saying no when the tooling drifted from them. The stack is Next.js with local-first SQLite: a single teacher's machine, no accounts, student names never sent to the API. Every live call is GPT-5.6 with a strict root-object Structured Output: extraction, full-page segmentation, diagnosis, grading proposals, practice, briefs, follow-up evaluations, predictions. Each call records prompt/schema versions, input/output hashes, token counts, and latency, and identical inputs are reused by hash before any new spend.

The test suite grew adversarially around failure modes: seventeen verify:* scripts covering faint ink, PDF signature abuse, ungrounded evidence, stale runs, legacy migrations, A4 print fragmentation, Prediction Lab invalidation. npm run check runs all of them plus lint, typecheck, and a production build before anything ships.

Challenges we ran into

Handwriting is hostile. In an early live test, a faint handwritten $=$ was read as a dash, which changes the mathematics entirely. Instead of tweaking prompts, that became engineering: an image pipeline with line-aware cropping and a full-frame fallback (7b061d0), the exact failing image kept as a permanent regression fixture, and a policy that caps confidence when a transcribed step is mathematically implausible.

Cheap was tempting. Benchmarking the available model tiers on a real six-page exam, the faster tier was about three times cheaper — and it failed the handwritten-equals test. Diagnosis stayed on the accurate tier. The full cost/quality matrix is in the repo (docs/model-benchmark.md, $0.83 for the 15-call benchmark).

Real data is humbling. I'm French, so the full national exam sets I could get were French brevet papers. I handwrote three student booklets per exam under invented names, each with its own error profile, and ran all five exams through the pipeline: 15 booklets, 393 questions diagnosed, 59 items flagged as uncertain rather than guessed, zero forced verdicts. And the Prediction Lab showed nothing for that class, no student repeated the same flawed rule on two distinct problems, so no model earned the right to predict. We left it empty rather than lower the evidence thresholds. A worse-looking demo, a more honest product.

What we learned

That abstention is a feature, not a failure mode. That pedagogy constraints belong in database triggers, where a refactor can't erase them. That within-student variability is real, exactly as Siegler & Pyke (2013) describe: a model with an observed application rate of 0.8 should expect 4 x 0.8 = 3.2$ hits out of four opportunities, and reporting 3-of-4 as "consistent" instead of "wrong" changes how a teacher trusts the tool.

What's next

Page-aware feedback markers on the original scans, cross-page matching for multi-page booklets, and a student mode for receiving corrected copies and doing the discrepant-event practice.

Built With

Share this project:

Updates