Pupil

Reading feels like understanding until the exam. Pupil is an AI student you teach, then test - revealing the gaps in your understanding through its mistakes.

Comment

Inspiration

We all know the feeling: you read a chapter, everything seems to make sense, and you feel like you understand it. Then the exam asks the question from a slightly different angle and suddenly you realize you never understood the concept as deeply as you thought.

This is closely related to what psychologists call the "illusion of explanatory depth": people can feel confident that they understand something until they are asked to actually explain it. Reading fluently can feel almost identical to understanding deeply, which makes our own gaps surprisingly difficult to see.

We noticed a problem with many existing AI study tools. They are designed to answer questions, explain concepts, and help students when they already know what they are confused about. But what happens when you don't even know what you don't understand?

The oldest solution to this problem is also one of the simplest: "teach it to someone else". Explaining an idea forces you to turn vague understanding into something concrete, and weaknesses begin to surface. The problem is that you rarely have a patient beginner available every time you study.

So we decided to build one.

What it does

"Pupil is an AI student that knows nothing, and it's yours to teach."

Instead of asking an AI tutor for answers, you write a lesson in your own words and teach Pupil a concept.

Pupil then builds an internal belief state based only on what you taught it. It identifies what it understood, what it only understood partially, and what misconceptions it still holds.

It then:

  1. "Shows you what landed." You can see the ideas the student actually took away, as well as misconceptions that survived.
  2. "Asks questions from inside its confusion." Pupil generates the questions a real beginner in that exact belief state would be likely to ask.
  3. "Sits an unseen exam." The AI student answers six physics questions using only the knowledge and beliefs it formed from your lesson.
  4. "Explains its mistakes." The Marking Table traces each wrong answer back to the belief that caused it and the weakness in your explanation that allowed that belief to survive.
  5. "Lets you reteach the gap." You can teach the weak concept again and watch the student's performance change.

The key insight behind Pupil is that there are different ways a lesson can fail.

A concept can be:

  • "Never mentioned" - the student's prior misconception remains untouched.
  • "Mentioned but not clearly taught" - the student retains the new fact while still holding the old misconception.
  • "Contradicted" - the student can end up believing the contradiction, and Pupil can quote the relevant sentence from the lesson back as evidence.

For example, a student can know the formula "F = ma" and still believe that force is what keeps an object moving. Pupil is designed to expose that kind of gap rather than simply telling the learner the correct answer.

That is the product: "we don't grade whether you got the answer right. We show you whether you actually taught the idea clearly enough for someone else to get it right."

How we built it

We built Pupil as a deliberately small hybrid AI system, combining language models with deterministic application logic.

The frontend uses "Next.js 16, React 19, TypeScript, and Tailwind CSS". The backend consists of two Next.js route handlers, and there is no database: application state stays on the client so there is less infrastructure that can fail during a live demo.

For AI, we built two interchangeable backends behind the same interface:

  • "Anthropic Claude Opus 5"
  • "NVIDIA NIM running Nemotron 3 Super 120B"

Both use the same prompts and the same Zod schemas, which means the two implementations follow the same expected structure and can be tested consistently.

The most important architectural decision was to keep the AI and the diagnostic logic separate.

Our pipeline is roughly:

"Your lesson → AI comprehension → belief synthesis → AI-generated beginner questions → AI exam → deterministic attribution"

A hand-authored concept graph represents Newtonian mechanics, with misconceptions based on the Force Concept Inventory. The AI maps the user's natural-language explanation onto those fixed concepts and evaluates whether each was taught, partially taught, or contradicted.

After that, deterministic TypeScript logic synthesizes the student's belief state and handles scoring and attribution. This allows the application to make its most important claims without asking an LLM to simply decide whether it thinks something is true.

The AI then operates inside that state: it generates questions from the student's confusion and answers an unseen exam according to those beliefs.

We also added validation with "Zod" at model-output and API boundaries, and our deterministic core currently has "67 passing assertions".

The result is a system where AI handles the messy, human part — understanding an imperfect lesson and simulating a beginner — while ordinary code handles the parts that need to be reproducible and explainable.

Challenges we ran into

1) The student kept being too good. Our first exam runs scored 4/6, and the reasoning showed why: it was using real physics knowledge we had never taught it. We hardened the prompt, and made the schema force it to quote the belief it was using before it was allowed to pick an answer. Field order in a JSON schema is generation order, so naming the belief first genuinely constrains what comes next.

2) But the real bug was ours, not the model's. Even after hardening, it scored 4/6. Re-reading its reasoning, the student was behaving correctly and our model of learning was wrong. We were treating a half-taught concept as clearing the misconception under it. The research says the opposite: partial instruction layers a fragile fact on top of a prior intuition without removing it. That is why real students recite "F = ma" and still think force means speed. Fixing the belief model took the score to 0/6 with every single miss attributable - and turned the biggest bug into the product's best feature.

3) Teaching more made the score go down. Every re-teach re-judged the whole transcript with a stochastic model, so adding a paragraph about friction re-litigated concepts it never mentioned. Teaching more must never unteach something. Now only the new lesson is judged and prior coverage is inherited.

4) A reasoning model that ran out of room. Nemotron burns its token budget on hidden reasoning and was truncating the JSON mid-object - which looks exactly like a model that cannot follow a schema. We made truncation a distinct, named error, then disabled the reasoning pass: comprehension went from 51 seconds to 8.

5) Choosing a model by measurement, not reputation. We wrote a harness that runs the real comprehension prompt against candidate models and scores each on whether it preserves the gaps a careful human marker would find. Two of six candidates were not available on our account at all, and one took 235 seconds per call.

Accomplishments that we're proud of

We are most proud that we built something fundamentally different from a conventional AI tutor.

Instead of making an AI that is always trying to be correct, we built an AI student that can be "wrong in a structured, diagnostic way". Its mistakes are not the end of the experience - they are the data that reveal where the teaching broke down.

We built a belief system capable of distinguishing between concepts that were never taught, partially taught, or contradicted. We connected those beliefs to an unseen assessment so that mistakes can be traced back to the belief that caused them and the weakness in the original explanation.

We are also proud of separating AI interpretation from deterministic diagnosis. Scoring, attribution, and recovery predictions are handled in tested TypeScript rather than being left entirely to an LLM. Our deterministic core currently passes "67/67 checks".

We also built safeguards around claims the product makes. Contradictions must be grounded in sentences that actually appear in the lesson, and recovery predictions are only made when the named concept is the sole identified obstacle.

Most importantly, we are proud that debugging changed the product itself. Our biggest early failure was discovering that our model of how people learn was wrong. Fixing that didn't just improve the implementation - it became the core insight behind Pupil.

What we learned

The most important thing we learned is that interesting AI failures are not always prompting failures.

We spent time trying to make the model behave better before realizing that part of the problem was our own model of learning. When an LLM behaves unexpectedly, sometimes the right question is not "How do we prompt it better?" but "Did we model the problem correctly in the first place?"

We also learned that an AI can sometimes be more useful when it is "reliably wrong" than when it is simply right.

Most educational AI tools are built around producing the correct answer. Pupil takes the opposite approach: a controlled wrong answer can reveal a misconception that a correct answer would hide.

We learned that trust comes from the deterministic layer. Once grading and attribution were handled entirely by a model, the results became difficult to verify. Moving those critical claims into testable code made the system more explainable and gave us much more confidence in what we were showing users.

Finally, we learned to only promise what the product can actually deliver. Our early recovery estimates were sometimes too optimistic. Making those predictions conservative made the feature smaller, but also made it more trustworthy.

What's next for Pupil

The current prototype focuses on Newtonian mechanics, but the underlying idea is not limited to physics.

The first major expansion would be "support for any subject". The concept graph and question bank are currently the hand-authored pieces, so the next step would be generating those structures automatically from a syllabus or learning material.

We also want to build a "classroom view" where a teacher could see the misconceptions appearing across an entire class and identify the concepts that are actually worth reteaching. Because Pupil already diagnoses understanding at the concept level, the next step would be aggregating those patterns across students.

Finally, we want to measure the educational effect with real learners. The pedagogy behind learning through explanation is well established, but we have not yet run a formal study of Pupil itself. A pre/post assessment with real students would allow us to move from a strong educational hypothesis to measurable evidence of learning gain.

Built With

Share this project:

Updates