Knowledge Dungeon

Turn your notes into a pixel-art dungeon: bosses quiz you from your own notes, and an ML model tracks what you're forgetting so the Dungeon Master can respawn it.

Comment

Inspiration

Every student knows the feeling: you re-read your notes the night before an exam, it all feels familiar, and a week later it's gone. Learning science has known the fix for decades: active recall (testing yourself) and spaced repetition (reviewing just as you're about to forget). Almost nobody sticks with them, because flashcard apps feel like homework.

Games, on the other hand, are great at getting people to come back. So I asked: what if your notes were the game? What if every topic was a room in a dungeon, every room had a boss that quizzed you, and the bosses you were starting to forget came back from the dead?

What it does

Knowledge Dungeon turns your notes, or a whole 100-page textbook, into a pixel-art dungeon crawler.

  • Upload a PDF, Markdown or text file. An LLM Dungeon Master finds the topics and how they depend on each other, and builds a map: one room per topic, locked until you've cleared the topics it builds on.
  • Fight each room's boss. Bosses ask multiple-choice questions written from your own notes, and every question is double-checked before you see it.
  • Remember. A knowledge-tracing model trained on 325k real student answers estimates how well you know each topic. It sets each boss's difficulty and HP, and when it predicts you're forgetting a topic you already cleared, the Dungeon Master respawns that boss and points you to it.

It's spaced repetition and adaptive practice, disguised as an RPG.

How I built it

1. The ML: a topic-agnostic knowledge tracer

Classic knowledge-tracing models like Deep Knowledge Tracing (DKT) learn a fixed list of skills. A model trained on maths problems can't say anything about "Deadlocks" from your OS notes, and in this game every player brings brand-new topics.

So instead of learning what the skills are, I trained a model on how people learn, using 12 features that exist for any topic: attempts, correct answers, accuracy on the topic, the last three outcomes, current streak, time since the topic was last practised, overall accuracy, recent accuracy, and how many topics the learner has seen. The model predicts

$$P(\text{correct} \mid \text{learner history on this topic})$$

with gradient boosting (scikit-learn's HistGradientBoostingClassifier). The same feature code builds the training rows and scores live players, so there's no gap between training and the game.

Trained and tested on ASSISTments 2009 (the standard DKVMN benchmark split):

Model Test AUC
Learner's overall accuracy (baseline) 0.682
Logistic regression, same features 0.725
Topic-agnostic tracer (used in the game) 0.759
DKT LSTM, skill-specific (reference) 0.819

I also trained a PyTorch DKT as a reference. It matches published DKT results, which shows the pipeline is sound, but it can't transfer to new topics. The tracer gives up some AUC to work on topics it has never seen, which is the trade-off this game needs. It trains in about 15 seconds on a laptop CPU.

In the game, the tracer's prediction \( m \) for a room drives the fight:

  • difficulty is easy if \( m < 0.5 \), medium if \( m < 0.75 \), otherwise hard
  • boss HP is \( 3 + \operatorname{round}\big(3(1-m)\big) \), so weaker topics mean longer fights and more practice

2. The agent: an LLM Dungeon Master with guardrails

After every fight, the Dungeon Master takes a turn as a tool-calling agent (native function calling) with four tools: get_map, respawn_room, recommend_room and finish. It respawns bosses when mastery drops below 65% or a spaced-repetition review is due (1 → 3 → 7 → 14 → 30 days), recommends the most valuable next room with a reason, and narrates in character.

The LLM decides how to act, but code enforces what must happen: finish is refused until fading rooms are respawned and a next room is recommended. Invalid tool calls are returned to the agent to fix, and if the LLM is unreachable, a rule-based policy makes the same kinds of moves, so the game never breaks.

3. Questions you can trust

For each room, TF-IDF retrieval pulls the relevant sections of your notes, and the LLM writes questions only from them. A second blind verifier pass answers each question without seeing the key, and any question where it disagrees, or that it flags as ambiguous, is thrown away. This caught real mistakes in testing, like a wrong page-table-size calculation.

4. Whole textbooks, not just a few pages

A 100-page book is far too big to send to an LLM on a free tier. So the app reads the PDF's structure: chapters from its outline and section headings from font sizes. The Dungeon Master then plans rooms from a compact chapter outline (~15k characters). Each room remembers its own chapter, and its questions come only from that chapter. A 10-chapter DBMS book becomes a 10-room dungeon in about 5 seconds.

5. Design

A retro desktop in white, cream, peach and ink. The sky is a drifting cumulus field rendered at low resolution on a canvas and ordered-dithered with a 4×4 Bayer matrix into the five-colour palette: each pixel's brightness \( L \) is snapped up or down depending on whether its fractional part beats the matrix threshold \( T_{x \bmod 4,\, y \bmod 4} \). Every panel is a little window with a striped title bar, news arrives in pop-ups ("You have received a dungeon."), and the 12 bosses are hand-drawn 16×16 sprites picked from the boss's name, so "The Data Golem" is a golem.

Stack: Python, FastAPI, SQLite, scikit-learn, PyTorch (benchmark), PyMuPDF, Groq (gpt-oss-120b) through the OpenAI-compatible API, Next.js, React and TypeScript. It's deployed on Render and Vercel.

Challenges I ran into

  • A tracer that could be right but feel wrong. The model sometimes lowered its estimate right after a correct answer. That can be statistically valid (e.g. it re-weights recent history), but it feels unfair to a player. I tried monotonic constraints, but AUC fell to 0.7435 and drops after correct answers still happened. So I kept the better model and added a display rule instead: during a fight, a correct answer never shows mastery going down, while the raw score still drives difficulty and respawns.
  • The LLM put the right answer at "B" 59% of the time, even when the prompt said to vary it. A player could just press 2. The lesson was not to trust the prompt for this: options are now shuffled in code, and the saved questions were re-shuffled, which evened out the answer positions.
  • A reasoning model that returned empty replies. gpt-oss spent its tokens thinking and returned nothing. The fixes were a lower reasoning effort, larger token limits, JSON mode for structured calls, and native function calling for the agent.
  • Free-tier rate limits (8,000 tokens/minute). Writing and verifying questions takes seconds, so the app prefetches questions in the background for the rooms you're likely to enter next. Fights now start in about 0.1 s.
  • Large PDFs were silently truncated. Early versions only used the first 60k characters, so a textbook became a dungeon of its first three chapters. Reading the PDF's own structure fixed this.

Accomplishments I'm proud of

  • A real ML model with honest evaluation against baselines and a published reference, chosen for the trade-off the product actually needs.
  • An agent with guardrails and a fallback, so the LLM adds personality without being able to break the rules.
  • It works on anyone's notes, from a one-page cheat sheet to a full textbook, and the demo dungeon works with no API key at all.

What I learned

  • Pick models for the product, not the leaderboard. DKT scores higher, but a model that works on unseen topics is the one that's actually useful here.
  • Don't trust an LLM with anything you can enforce in code: answer positions, required agent actions, prerequisite order.
  • Verification is cheap insurance. A second, blind LLM pass catches mistakes a student would otherwise "learn".

What's next

  • Accounts and a persistent database, so progress follows you across devices.
  • Letting players flag a bad question, and feeding those flags back into the verifier.
  • Fine-tuning the tracer on in-game play, so it adapts to how people learn from their own notes.
  • Free-response and diagram questions, not just multiple choice.

Built With

Share this project:

Updates

Submission history