Inspiration
Recognition isn’t understanding, and I used to think it was, honestly, for most of my life as a student. You read a paragraph, nod, feel the “yeah I get it” click, and move on. But that click is cheap, it’s pattern-matching against familiar tokens, not reconstruction. The only real test of comprehension is generative: can you rebuild the idea in your own language without leaning on the source? Self-explanation prompting isn’t a vague pedagogical hunch either, it has a meta-analytic effect size of $g \approx 0.55$ across roughly 64 studies and $n \approx 6,000$ learners. That’s one of the more reliable levers in the entire learning-sciences literature, and yet almost nothing shipped checks a student’s free-text explanation against the source they claim to understand. Every AI study tool explains to the student. I flipped it.
What it does
Input: 2-3 paragraphs of source material S, plus a student explanation E. Output: a four-state diagnostic map ϕ: E → {green, yellow, red, grey} applied per-claim, not globally.
- Green — claim stated, justified against S
- Yellow — claim stated, justification span = ∅
- Red — claim contradicts S
- Grey — confidence below threshold, system abstains rather than guesses
Every non-green flag carries a literal source anchor (the sentence in S that triggered it) plus a revision hint. Ends with one follow-up question targeted at the actual gap set, not a generic quiz. Formative only. No scores, no accounts, no storage. The loop is the product, not the grade.
How I built it
Three-call pipeline:
- Call A — extract atomic claims {c₁, ..., cₙ} from E
- Call B — for each cᵢ, return justification spans Jᵢ ⊆ S or Jᵢ = ∅
- Call C — verify claim-to-concept alignment, batched, one request for all propositions
Call B is the highest-risk component in the whole system, because the default LLM failure mode is helpfulness: given a bare unjustified claim, the model wants to supply the justification from S itself, silently mapping every yellow to green and gutting the entire product thesis. Hard invariant: verbatim-span grounding or discard, Jᵢ is never a paraphrase. Hardened via few-shot examples anchored on Jᵢ = ∅ cases.
Concept matching runs on bge-small-en-v1.5 embeddings with cosine similarity cos(θ), thresholded as:
- ϕ(cᵢ) = clear if cos(θ) ≥ T_HIGH
- ϕ(cᵢ) = grey if T_LOW ≤ cos(θ) < T_HIGH
- ϕ(cᵢ) = off if cos(θ) < T_LOW
T_HIGH and T_LOW are derived empirically, not borrowed, from a hand-labeled 20-pair dev set (7 clear / 7 partial / 6 off), set as percentile cuts:
- T_HIGH = P₁₀(clear)
- T_LOW = P₂₅(partial)
Cosine values are not comparable across encoders, so importing a threshold from a paper is just wrong. I built a specificity gate on top of this after empirically finding an 8:1 false-positive rate on red flags at realistic error rates—not a UX nicety, a necessary correction. Frontend renders sorted, non-overlapping span highlights; grey uses a dashed underline specifically to encode “uncertain,” not just a duller hue.
Challenges I ran into
The Call B helpfulness leak was the hardest problem in the build, full stop. Models want to be nice. Niceness is the enemy of an honest comprehension check. I only caught it by building a golden regression test, 10 hand-labeled samples with expected per-claim states, re-run after every prompt change from that point forward:
Agreement = (1 / n) * ∑ 𝟙[ϕ̂(cᵢ) = ϕ*(cᵢ)] ≥ 0.80
Skip that gate and you won’t notice your grader quietly degrading. Second fight: threshold calibration, where clear and partial distributions initially overlapped. The fix wasn’t the threshold math, it was the embedding input: switching to label + ". " + anchor instead of the label alone is what separated the distributions. Small change, real diagnostic cost to find.
Accomplishments that I’m proud of
Shipping a calibrated, source-anchored, deterministic pipeline in a 5-day window—that’s the actual accomplishment, not any single feature. Anyone can wrap an LLM call and call it a study tool. I hand-labeled data, tuned T_HIGH / T_LOW against real distributions, and gated on regression tests before touching UI polish. The grey state is, I think, the single most defensible design decision in the project: it’s a formal admission of P(uncertain) > 0, and that kind of honesty is rarer in this space than it should be.
What I learned
I used to think the hard part of an AI education tool was the AI part. It’s not. It’s calibration discipline, refusing to let the model cut corners toward false confidence just because false confidence reads as more helpful. The real differentiator against “just use ChatGPT” was never raw capability—a sufficiently careful prompt could approximate this in a single turn. It’s productization: deterministic, calibrated, reproducible, instead of a one-off prompt 99% of students will never write correctly.
What’s next for Explain-Back
- Expand the misconception dictionary beyond the current single-domain seed set (membrane transport, 8-10 entries)
- Multi-source support, currently intentionally scoped out
- Tighten T_HIGH / T_LOW against a larger labeled set ($n=20$ is a floor, not a target)
- Stress-test Call B against adversarial student inputs designed to bait false justification
Log in or sign up for Devpost to join the conversation.