Marginalia 📚🧠 — The AI Learning Engine
Private, source-grounded personal AI learning engine powered by NVIDIA Nemotron models on Nebius Token Factory. Never hallucinate. Never forget what you know. Never lose your place in your sources.
💡 Inspiration
Every student, researcher, and self-directed learner knows the frustration of drowning in 500-page textbooks, dense research PDFs, and technical slide decks. While conversational AI chatbots promised a revolution in study habits, standard LLMs suffer from three fatal flaws when applied to real learning:
- Hallucination & Fabricated Citations: Generic chatbots answer authoritatively but hallucinate non-existent page numbers, invent papers, or subtly alter technical theorems.
- Cognitive Amnesia (Zero Student Modeling): Standard chats treat every session as day one. They have no concept of what you have mastered versus what you struggle with, lacking mathematical learner tracking and spaced repetition.
- Passive Summarization vs. Active Recall: Generic AI generates passive walls of text instead of diagnosing conceptual misunderstandings, probing cognitive boundaries, or compiling structured pedagogical lessons.
We built Marginalia to solve this. Marginalia is not a chatbot; it is a closed-loop pedagogical learning engine that transforms your raw notes and documents into an active, mathematically modeled curriculum: $$\text{Ingest} \longrightarrow \text{Understand} \longrightarrow \text{Practice} \longrightarrow \text{Measure} \longrightarrow \text{Schedule} \longrightarrow \text{(Repeat)}$$
🚀 What Marginalia Does
Marginalia is built around five screenshot-verifiable, deterministic capabilities:
- Strictly Grounded or Silent: Every factual statement cites exact chunk IDs and 1-based page numbers. Quotes are strictly verified as verbatim substrings against the original text. If your uploaded notes do not contain sufficient evidence to answer a question, Marginalia honestly refuses (
abstention: true) rather than guessing. - Hybrid Dense Vector Retrieval with Noise Gating: Fuses SQLite FTS5 BM25 lexical search with dense float embeddings stored as IEEE-754 binary blobs (
chunk_embeddings), routed through Reciprocal Rank Fusion (RRF with ( k = 60 )) and dynamic noise thresholds (( 0.05 ) / ( 0.20 )) for zero-keyword semantic recall. - Autonomous Prerequisite Knowledge Graph: Automatically extracts a grounded Directed Acyclic Graph (DAG) with PageRank centrality, connected-components clustering, and topological milestones (Cytoscape + Mermaid.js).
- 5 Pedagogical Explanation Modes: Translates dense material into 5 cognitive styles: Simple (ELI5), Standard, Technical with KaTeX math, Source-Grounded, and Bangla (বাংলা), tagged with epistemic provenance badges (
SOURCE,HELPER,INFERENCE). - Cognitive Learner Modeling (Bayesian Knowledge Tracing): Tracks concept mastery ( p(\text{known}) ) in real time, updating dynamically on every answer attempt: $$p(L_{t+1} | \text{obs}) = \begin{cases} \frac{p(L_t)(1 - P(S))}{p(L_t)(1 - P(S)) + (1 - p(L_t))P(G)} & \text{if correct} \ \frac{p(L_t)P(S)}{p(L_t)P(S) + (1 - p(L_t))(1 - P(G))} & \text{if incorrect} \end{cases}$$
- Modern Spaced Repetition (FSRS): Implements the Free Spaced Repetition Scheduler computing memory stability ( S ), difficulty ( D ), and power-law retrievability: $$R(t) = \left(1 + \frac{t}{9S}\right)^{-1}$$
- Multi-Round Socratic Teach-Back & Misconception Diagnosis: Students explain concepts back in their own words (in English or Bangla); Marginalia isolates cognitive flaws, cites correcting passages, and stores them in persistent misconception memory until proven resolved across two consecutive trials.
- Agent Interoperability via Model Context Protocol (MCP): Exposes an official FastMCP server (
app/mcp/server.py) overstdiowith 8 core study tools, allowing Claude Desktop or Claude Code to act as an external tutor querying the local learner model.
🛠️ How We Built It
Marginalia pairs high-performance local-first system engineering with specialized NVIDIA Nemotron model tiers hosted on Nebius Token Factory:
1. NVIDIA Nemotron Model Tier Routing Matrix
Rather than relying on a single slow model, Marginalia routes tasks dynamically across specialized Nemotron tiers based on latency and reasoning requirements:
| Pipeline Stage | Model on Nebius Token Factory | Measured Latency | Selection Rationale |
|---|---|---|---|
| Reasoning Tier | nvidia/nemotron-4-340b-instruct |
~2.4s | Deep conceptual evaluation; distinguishes between surface phrasing variance and genuine cognitive misconceptions during Teach-Back. |
| Generation Tier | nvidia/llama-3.1-nemotron-70b-instruct-hf |
~1.1s | High adherence to strict negative constraints ("abstain if not in context", verbatim citation extraction, semantic rubric grading). |
| Fast Extraction Tier | nvidia/nemotron-mini-4b-instruct |
~320ms | High-throughput parallel ingestion (asyncio.Semaphore(3)); extracts atomic definitions and prerequisite links without bottlenecking uploads. |
| Embedding Tier | nvidia/nv-embedqa-e5-v5 |
~85ms | Generates 256-d normalized semantic vectors stored as SQLite IEEE-754 binary blobs. |
2. The Core Architecture
- Backend: Python 3.12, FastAPI, Pydantic v2, SQLite WAL mode with FTS5 BM25, and a custom IEEE-754 vector similarity engine.
- Frontend: React 19, TypeScript, Tailwind CSS, KaTeX (LaTeX math rendering), Cytoscape.js (DAG visualization), and Mermaid.js (procedural visual compiler).
- Evaluation Framework: Automated regression evaluation suite (
evals/run_evals.pyonevals/gold.jsonl) calculating Recall@5 and Mean Reciprocal Rank (MRR). - Agent Ecosystem: Model Context Protocol (FastMCP) server over
stdioexposing 8 native study tools. - Privacy & Security: Local-first data storage, POSIX
0600credential file permissions, and SSRF boundary defense blocking RFC 1918 private subnets and cloud metadata endpoints (169.254.169.254).
⚡ Challenges We Faced
- Eliminating Subtle Citations Hallucination: Standard LLMs tend to paraphrase or guess page numbers. We engineered a two-stage verification harness: the model extracts candidate citations, and our backend performs deterministic substring matching against the source chunk database, rejecting any citation that is not 100% verbatim.
- Dense Vector Storage Without Heavy External Vector DBs: To preserve the privacy and simplicity of a zero-setup local application, we avoided bulky external vector databases. Instead, we serialized 256-dimensional embeddings as raw IEEE-754 binary blobs directly inside SQLite, implementing high-throughput cosine similarity fused with BM25 via Reciprocal Rank Fusion (( k = 60 )).
- Cross-Lingual Technical Ingestion (Bangla + English): Many bilingual students read textbooks in English but express doubts or take notes in their native language. Achieving zero-shot cross-lingual grounding required robust query rewriting and schema defenses to prevent language confusion during Socratic teach-back grading.
- Latency Tradeoffs in Educational Workflows: Running 340B parameter models for every interaction would make the app sluggish. Splitting tasks into parallel 4B extraction, 70B generation, and reserved 340B reasoning gave us sub-second responsiveness while preserving world-class pedagogical evaluation.
🏆 Accomplishments That We're Proud Of
- Zero-Hallucination Grounding: In benchmark testing over our evaluation corpus, Marginalia achieved 0 unverified citations and cleanly abstained on out-of-domain queries.
- Empirical Retrieval Superiority: In our ablation benchmark (
evals/gold.jsonl), our hybrid RRF retrieval achieved a +26.67% Recall gain over pure BM25, hitting 1.0000 MRR and 1.0000 Recall@5. - Adaptive Learning Acceleration: In synthetic learner simulations (( N=100 ) learners, 20 practice trials), proximal item selection targeting ( p \approx 0.70 ) achieved a mean mastery gain of ( \Delta\text{BKT} = +0.5595 ) compared to stagnation under random selection (( \Delta = -0.0513 )).
- Rock-Solid Engineering Gate: Passed all 270 unit & integration tests (100% pass rate), 19 Playwright end-to-end browser tests, strict Mypy typing across 59 source files, and automated secret/SSRF security audits.
- FastMCP Server Support: Native compatibility with Claude Desktop and Claude Code out of the box.
📖 What We Learned
- Model Tiering is the Future of Personal AI: Using a massive model for everything is wasteful and slow; using a tiny model for everything degrades reasoning. Tiering models by task complexity (Nemotron 4B for extraction, 70B for generation, 340B for cognitive diagnosis) produces an experience that feels instantaneous yet deeply intelligent.
- Learning Science Needs Math, Not Just Text: LLM chat transcripts are terrible for studying because they have no model of human memory. Pairing LLMs with established cognitive science algorithms—Bayesian Knowledge Tracing and FSRS spaced repetition—converts an AI from an entertaining novelty into a genuine learning engine.
🔮 What's Next for Marginalia
- Multi-Modal Lecture Video & Audio Ingestion: Automatic transcription and slide OCR with frame-accurate timestamp citations.
- Collaborative Study Groups via MCP: Federated peer-to-peer knowledge graph sharing without exposing raw private notes.
- Offline-First Local Acceleration: On-device quantizations of Nemotron-Mini for offline mobile and edge study sessions.
Log in or sign up for Devpost to join the conversation.