StudyPilot: Adaptive Multimodal Tutor

Unified multimodal tutor that cites textbooks, slides, and lectures with zero hallucinations—powered by BKT and spaced repetition.

Comment

What's next for StudyPilot: Adaptive Multimodal Tutor

StudyPilot — Adaptive Multimodal AI Study Companion Multimodal AI Hackathon 2026 · Track D: Personalized Tutoring & Adaptive Learning

💡 Inspiration Every semester, university students face the exact same cognitive overload:

Fragmented Course Materials: 600-page textbook PDFs, 40-slide PPTX decks with obscure diagrams, and 90-minute lecture video recordings scattered across different portals. Hallucination Anxiety: Generic LLMs often produce plausible-sounding explanations that contradict the professor's syllabus or invent facts when solving domain-specific exam questions. One-Size-Fits-All Practice: Static quiz questions fail to adapt to a learner's actual cognitive state, either boring them with trivial problems or demoralizing them with questions far beyond their current ability. We asked ourselves: What if an AI tutor could ingest every textbook, slide, and lecture recording into a single unified knowledge base, cite the exact page, slide, or video timestamp for every single sentence it utters, and adaptively calibrate assessments using cognitive science models?

That vision is StudyPilot.

🚀 What It Does StudyPilot transforms heterogeneous academic materials into an interactive, 100% source-grounded learning companion:

Unified Multimodal Grounding: Ingests textbooks (PDF), slide decks (PPTX), and lecture recordings (MP4/YouTube) into a shared vector knowledge base. Every answer and explanation carries verifiable citations like [Textbook, p.14], [Slide 12], or [Lecture @ 14:32]. Zero-Hallucination Citation Inspector: Clickable citation badges open an interactive inspector displaying the raw text chunk extracted from the verified course materials. Multimodal Chat Tutor with 4 Pedagogical Modes: Real-time streaming tutor that switches between Explain Simply, Deep Dive (Rigorous Math), Analogy Mode, and Exam Summary. Adaptive Assessment Engine: Dynamically selects questions that target the student's Zone of Proximal Development rather than random sampling. Cognitive Mastery Modeling: Tracks latent student mastery per concept using Bayesian Knowledge Tracing (BKT), dynamically updates difficulty with Elo ratings, and schedules reviews using SuperMemo-2 (SM-2) spaced repetition curves. Personalized Learning Dashboard: Visualizes topic mastery heatmaps, concept prerequisite dependency graphs, and automatically builds an hour-by-hour day-by-day exam prep schedule. 🛠️ How We Built It StudyPilot was engineered with a modern full-stack decoupled architecture designed for high throughput and low latency:

  1. Ingestion & Retrieval Pipeline PDF Extraction: Built with PyMuPDF to preserve document hierarchy, page numbers, and embedded figures. Presentation Slides: Parsed with python-pptx, capturing slide titles, bullet points, and speaker notes with per-slide indexing. Video & Audio Transcription: Utilizes yt-dlp and faster-whisper to transcribe audio streams into timestamped sentence segments. Visual Diagram Captioning: Integrated with Gemini Vision to generate dense semantic descriptions for charts, diagrams, and formulas. Hybrid Retrieval: Chunks (~500 tokens with 100-token overlap) are embedded using Gemini embedding models and indexed in ChromaDB. At query time, we perform hybrid search combining dense vector cosine similarity (60% weight) with BM25 keyword reranking (40% weight).
  2. Cognitive Science Algorithms Bayesian Knowledge Tracing (BKT) We model whether a student has mastered latent skill $L$ at step $t$. When the student answers a question with outcome $C_t \in {0, 1}$, the posterior mastery probability is updated via Bayes' rule:

If the answer is correct ($C_t = 1$): $$P(L_t \mid C_t = 1) = \frac{P(L_t) \cdot (1 - s)}{P(L_t) \cdot (1 - s) + (1 - P(L_t)) \cdot g}$$

If the answer is incorrect ($C_t = 0$): $$P(L_t \mid C_t = 0) = \frac{P(L_t) \cdot s}{P(L_t) \cdot s + (1 - P(L_t)) \cdot (1 - g)}$$

where:

$s = 0.10$ is the slip probability (knowing the concept but making a mistake) $g = 0.20$ is the guess probability (not knowing the concept but guessing correctly) Before the next problem, mastery accounts for learning transition probability $P(T) = 0.15$: $$P(L_{t+1}) = P(L_t \mid \text{obs}) + (1 - P(L_t \mid \text{obs})) \cdot P(T)$$

Elo Dynamic Difficulty Adjustment Both the learner rating $R_{\text{learner}}$ and question difficulty $R_{\text{item}}$ update dynamically with $K = 32$: $$E = \frac{1}{1 + 10^{(R_{\text{item}} - R_{\text{learner}}) / 400}}$$ $$R_{\text{learner}}^{\text{new}} = R_{\text{learner}}^{\text{old}} + K \cdot (\text{Score} - E)$$

SuperMemo-2 (SM-2) Spaced Repetition Next review dates are projected according to memory stability intervals: $$I(n) = \begin{cases} 1 \text{ day}, & n = 1 \ 6 \text{ days}, & n = 2 \ I(n-1) \cdot EF, & n > 2 \end{cases}$$

  1. Frontend & Full-Stack Tech Stack Frontend: Next.js 16 (App Router) + TypeScript + Tailwind CSS with glassmorphism styling, Lucide icons, and Recharts for interactive mastery heatmaps. Backend: FastAPI (Python 3.11), SQLAlchemy, SQLite, ChromaDB in-process vector store. LLM Engine: Gemini 2.5 Flash for high-speed streaming tutoring, question generation, and rubric-based grading. 🧗 Challenges We Faced Cross-Modal Alignment: Mapping text from a PDF textbook to a 3-second audio snippet in a 90-minute lecture video was non-trivial. We solved this by developing a standardized citation schema (SourceChunk metadata) that binds relative locations (page numbers, slide indices, start/end timestamps) into every retrievable embedding unit. Preventing LLM Hallucinations in Citations: Early prototypes sometimes generated plausible citations to non-existent pages. We implemented a strict grounding verification filter: the LLM is only provided retrieved chunk IDs and metadata, and our post-processing layer cross-validates every citation tag against SQLite ground truth before rendering. Real-Time Streaming with Dynamic Grounding: Generating SSE streams while parsing structured citation metadata required an asynchronous generator pipeline that streams text tokens smoothly without breaking citation bracket parsing. 🎓 What We Learned Cognitive Science + LLMs is a Superpower: While LLMs are exceptional at generating explanations, classical educational algorithms like BKT and SM-2 provide the deterministic mathematical grounding necessary to track student progress reliably over time. Multimodal Context Density: Transcribing video lectures produces significantly noisier text than curated textbook PDFs; combining BM25 keyword matching with dense vector embeddings dramatically improved recall accuracy on lecture transcripts. Explainability Builds Learner Trust: Adding the "Why this question?" badge and the interactive citation inspector transformed the app from a "black-box quiz bot" into a transparent learning companion students actually trust. 🔮 What's Next for StudyPilot Audio Tutoring Voice Mode: Conversational voice interactions with real-time interruptions during study sessions. Collaborative Peer Knowledge Graphs: Allowing study groups and university cohorts to merge course knowledge bases and compare mastery analytics. Multi-Course Graph Navigation: Cross-referencing concepts across semesters (e.g. connecting Linear Algebra prerequisites directly to Deep Learning architectures).

Built With

Share this project:

Updates

Submission history