Inspiration
Teachers get exit ticket scores back and can see who missed question 4. Working out why takes longer than the prep period they have. I wanted to build the thing that answers why, and then I wanted a number telling me how often it was right.
What it does
Takes a CSV of a class's quiz responses. Maps each wrong answer to a named misconception from the Eedi taxonomy (2,587 labels), then aggregates a class profile ranked by how many students each misconception affects and writes a reteach plan.
Then it builds a recheck quiz. GPT-5.6 writes the stem, SymPy computes the correct answer, and an executable buggy-math rule generates the distractor. Every finished item goes back through diagnosis blind. Items that fail to map to their target misconception are rejected.
Output: a static HTML class report, a Markdown reteach plan, and a verified recheck quiz.
How I built it
Codex built the core over three days: benchmark harness, executable rules, factory, report renderer. GPT-5.6 runs at inference time for three cached misconception hypotheses, rerank and verification over a fixed local candidate pool, stem extraction, factory item generation, and reteach writing.
Retrieval is local: BGE-large hybrid plus a TF-IDF arm, combined with reciprocal-rank fusion. That keeps the baseline free and lets the GPT-5.6 contribution be measured on its own. SymPy grounds every computable answer. GPT responses are cached by model, prompt, and reasoning effort, and the cache is committed, so anyone can regenerate the report and the metrics without an API key.
All of this ran on gpt-5.6-luna. I picked the cheap model on purpose: the whole point is a teacher running this on every exit ticket for every class, and that only works if one class costs cents. The lift comes from the retrieval stack: three cached hypotheses, hybrid retrieval, rerank and verify. I never benchmarked terra, so I don't know whether the bigger variants do better.
Challenges I ran into
The retrieval-only baseline wasn't actually retrieval-only. use_rerank=False skipped the rerank call but still ran the GPT-5.6 hypothesis arm, so the number I was calling "local embeddings alone" had GPT in it. That inflates the lift, which is the entire headline claim. I added an early return so the flag kills the whole GPT path, wrote a guard test for that boundary, then replayed all 716 held-out items with CLASSLENS_OFFLINE=1 and no GPT client instantiated. It came back 54/716 @1 and 143/716 @3, matching the published 7.5% and 20.0%. The lift was clean, but I had no way to know that until I could run the baseline with the client physically absent.
Accomplishments that I'm proud of
Diagnosis on 716 held-out real Eedi pairs: local retrieval 7.5% @1, GPT-5.6 best arm 41.1% @1 and 55.4% @3. A +33.5-point lift, 95% CI +27.8 to +38.9.
Executable rules reproduce the exact distractor a real student picked 71 times out of 258 (27.5%), with parse failures counted in the denominator.
The factory accepted 10 of 33 items and rejected 23, with the rejection reason logged for each one and shown in the report.
What I learned
Writing the benchmark before tuning against it killed several changes that felt like wins. A k=50 candidate pool didn't beat k=25 on recall@25. A high-effort selector variant didn't move accuracy@1. Both would have shipped on intuition.
Rejection has to be visible or verification is decorative. The factory only became worth anything once the 23 failures sat in the report next to the 10 successes.
What's next for ClassLens
Grammar coverage is what caps distractor reproduction. 41.1% of eligible items parse, and 67.0% of those reproduce correctly, so widening the parser moves the headline without touching a single transform. After that, more Algebra-1 rules, then geometry, and an importer for LMS quiz exports.
Log in or sign up for Devpost to join the conversation.