LearnProof
The Problem
Most AI learning tools are good at generating questions, but much weaker at answering the more important question:
Why does a learner keep getting something wrong?
A wrong answer can come from a misconception, a missing prerequisite, a shallow understanding, or simply low confidence. Treating every wrong answer as the same problem leads to generic remediation and little evidence that the learner actually improved.
We built LearnProof to investigate the learner rather than simply grade them.
The Idea
LearnProof turns a textbook or other structured source document into a source-grounded learning environment.
A learner selects a topic from the source's real instructional structure. LearnProof identifies the bounded content belonging to that topic, retrieves relevant evidence from the source, and uses an LLM to create diagnostic questions around what was actually taught.
The learner then moves through:
Wrong → Why? → Repair → Proof
The goal is not just to get the next answer correct.
The goal is to determine whether the learner has actually understood the concept and can transfer that understanding to a new situation.
How It Works
1. Source
The user uploads a textbook or source document.
LearnProof uses Microsoft MarkItDown to convert the source into machine-readable Markdown while preserving the document as the canonical knowledge source. Page-aware processing retains the connection between extracted content and its original source pages.
The document's real table of contents is used to reconstruct its instructional hierarchy:
Textbook
└── Main topic / chapter
├── Subtopic
├── Subtopic
└── Subtopic
This means LearnProof does not require a subject-specific question bank.
A Grade 8 Science book and an Astronomy textbook can produce completely different curricula from their own source structures.
2. Retrieval
The extracted content is chunked and stored in ChromaDB with source metadata such as page, main topic, and subtopic boundaries.
Semantic embeddings using all-MiniLM-L6-v2 allow LearnProof to retrieve relevant passages from the selected topic.
The selected topic establishes a hard content boundary, while its subtopics guide retrieval so that questions are based on the material actually taught across that topic rather than on a single nearest-neighbour passage.
3. AI Assessment
The retrieved source evidence is provided to an LLM to generate multiple diagnostic questions.
The questions are intentionally different in purpose:
- conceptual understanding
- application and reasoning
- misconception discrimination
The system is designed to avoid generic source-verification questions and instead generate questions that test the actual concepts, relationships, processes, and distinctions taught in the selected topic.
4. Learner Model
LearnProof does not use the answer alone.
Each attempt also includes the learner's confidence.
A lightweight Logistic Regression calibration model estimates the probability that the learner will answer the next question correctly using features such as confidence, difficulty, and calibration gap.
An adaptive risk score then helps determine which learning signal deserves investigation.
This gives us an important distinction:
A confident wrong answer is often more informative than a hesitant wrong answer.
5. Diagnosis
LearnProof examines the learner's response against the specific source evidence used by the question.
It distinguishes among:
- Possible misconception
- Knowledge gap
- Stable understanding
- Underconfidence
It can also trace prerequisite relationships between topics.
This makes the diagnosis specific to the learner's actual response instead of generating a generic explanation for the whole chapter.
6. Repair
Once the gap is identified, LearnProof generates a targeted repair.
The repair is grounded in the same source evidence that supported the diagnosis.
It can use formats such as an analogy, story, or exam-style explanation, while focusing on the exact learning gap rather than reteaching the entire textbook.
7. Transfer Verification
The final step is the most important.
LearnProof generates a new scenario that requires the learner to apply the repaired concept without simply repeating the original question or explanation.
A correct transfer answer provides stronger evidence that learning actually occurred.
The system therefore measures:
Before → Repair → Transfer
rather than simply:
Wrong → Correct
Why We Built It This Way
One of our biggest design challenges was avoiding hardcoded subject-specific logic.
Our first iterations could accidentally make a topic such as Oceanography retrieve an unrelated question bank. That was useful as a prototype, but it was not a genuinely versatile learning system.
We redesigned the pipeline around the source itself.
The document defines the curriculum.
The curriculum defines the scope.
ChromaDB provides the evidence.
The LLM generates and interprets the assessment.
ML models the learner.
This separation allows the same architecture to work across different subjects without replacing the system with a collection of subject-specific rules.
What We Learned
Building LearnProof taught us that using an LLM for education is not simply a matter of asking it to “generate a quiz.”
The difficult part is controlling what the model is allowed to reason about.
We had to make the source boundaries explicit, preserve page-level provenance, prevent unrelated retrieval, connect questions to evidence, and make diagnosis and repair operate on the exact concept being tested.
We also learned that confidence is valuable educational information. A learner who answers incorrectly with 95% confidence presents a very different investigation signal from a learner who answers incorrectly with 25% confidence.
Challenges
The hardest technical problems were:
Source understanding: textbooks contain covers, front matter, tables of contents, chapter structures, summaries, exercises, appendices, and administrative material that should not all become “concepts.”
Scoped retrieval: a long textbook can contain the same terminology in many places, so retrieval must respect the selected topic's content boundary.
Question quality: generated questions must test understanding rather than simply restating retrieved text.
Safe fallback behaviour: when an LLM is unavailable, the system should degrade gracefully without silently falling back to an unrelated hardcoded question bank.
Transfer validity: the final question must test genuine application rather than memorisation of the repair.
The Result
LearnProof combines:
Document understanding + semantic retrieval + LLM reasoning + adaptive ML + learning transfer verification
into a single loop.
We are not trying to build another AI quiz generator.
We are building an AI learning investigator:
Discover why → teach why → prove why.
Log in or sign up for Devpost to join the conversation.