-
-
cover photo
-
dashboard
-
multimodal_ingestion
-
grounded_tutor_chat
-
refusal_mechanism
-
adaptive_assessment
-
live_quiz interface
-
assessment_result
Inspiration
Modern students study across fragmented materials: PDF textbooks, presentation slide decks, and recorded lecture videos. General-purpose AI chatbots frequently produce hallucinations untethered from actual course materials and lack insight into what the student has already mastered.
Our goal for Track D: Personalized Tutoring & Adaptive Learning was to eliminate these hallucinations by building a multimodal companion that:
- Ingests all course formats without manual preprocessing.
- Explains concepts exclusively with exact page, slide, or video timestamp citations.
- Automatically refuses queries that fall outside the course syllabus.
- Models learner progress using Bayesian Knowledge Tracing (BKT) to tailor quizzes and eliminate repeated questions. ## What it does • Multimodal Knowledge Ingestion: Ingests textbooks (PDF with image/figure descriptions via Gemini Vision), slide decks (PPTX), and lecture recordings (temporal markers) automatically into an HNSW cosine vector index (ChromaDB). • Source-Grounded AI Tutor: Answers student questions with strict inline citations: [Source N: , Page X / Slide Y / timestamp MM:SS]. • Out-of-Curriculum Refusal: Detects queries absent from course material and declines to answer them, keeping study sessions strictly on syllabus. • Adaptive Assessment Engine: Generates MCQ, numerical, and short-answer problems on student-selected topics. Avoids repeated questions through MD5 cryptographic fingerprinting. • Learner Model (BKT): Updates probability of mastery P(Lₜ) per topic after every quiz response, dividing topics into Strengths (≥ 80%) and Weak Topics (< 60%) with tailored review suggestions. • Unified Analytics Dashboard: Visualizes document chunks, active quizzes, and longitudinal mastery curves. ## How we built it • Backend: FastAPI (Python 3.11) with SQLite metadata storage and ChromaDB for dense vector search. • Multimodal AI: Google Gemini 2.0 Flash (gemini-2.0-flash) for fast reasoning and vision, paired with models/text- embedding-004 (768-dim embeddings). • Document Extractors: PyMuPDF (fitz) for PDFs and diagram descriptions, and python-pptx for presentation decks. • Learner Modeling: Custom implementation of Bayesian Knowledge Tracing with P(L₀) = 0.30, P(T) = 0.10, P(S) = 0.10, and P(G) = 0.25. • Frontend: Ultra-responsive Single Page Application (SPA) built with Alpine.js and Tailwind CSS. • Cloud Infrastructure: Containerized with Docker and deployed globally on Fly.io edge machines on internal port 8080. • Evaluation: Evaluated with custom RAGAS-equivalent benchmark suites and simulated student trajectory agents. ## Challenges we ran into • Multimodal Grounding Consistency: Ensuring diagrams and visual tables in textbooks were accurately represented in the vector store required rasterizing page regions and synthesizing visual descriptions. • Question Deduplication Across Sessions: Simple question matching fails when LLMs rephrase prompts. We implemented a canonicalization hashing protocol to eliminate question repetition across consecutive quiz sessions (achieving a 0.0% repetition rate). • Deployment Isolation: Configuring the application to run smoothly on port 8080 on Fly.io without conflicting with existing services. ## Accomplishments that we're proud of • Achieved 0.9886 Faithfulness and 100% Out-of-Curriculum Refusal Accuracy on our benchmark test suite. • Demonstrating a +69.0% mastery gain in simulated student trajectories across 5 learning iterations. • Generating a self-contained, live-hosted full-stack platform and complete 1080p demonstration video. ## What we learned • Multimodal Grounding is More Than Raw OCR: Simply converting lecture slides or textbook pages into flat text strips away essential visual context (e.g., biological cycle diagrams, system architectures, chemical reactions, geometric curves). Augmenting extracted page images with vision-language descriptions via Gemini Vision prior to embedding significantly boosted retrieval faithfulness (0.9886) and context recall (1.0000). • Probabilistic Modeling Outperforms Static Heuristics: Traditional quiz systems either use random selection or naive difficulty tiers. Integrating Bayesian Knowledge Tracing (BKT) provided mathematical confidence in learner state transitions, allowing the companion to naturally account for accidental slips (P(S) = 0.10) and lucky guesses (P(G) = 0.25) while producing measurable mastery gains (+69.0% across 5 learning iterations). • The Value of Explicit Guardrail Rejections: General LLMs default to trying to answer every prompt, even when the topic is out of syllabus. Enforcing a strict refusal protocol for out-of-curriculum queries (achieving 100% refusal accuracy on our benchmark) was essential to guaranteeing that the study companion remains a dependable, syllabus- grounded academic tool rather than a source of distraction or hallucination. • Question Deduplication Requires Normalization: LLMs frequently regenerate functionally identical questions with minor phrasing alterations. Establishing a cryptographic fingerprinting pipeline across historical student records is critical for eliminating repetitive questions across continuous study sessions (achieving a 0.0% question repetition rate). ## What's next for StudyCompanion AI – Source-Grounded Multimodal Tutor • Automated Visual Concept & Prerequisite Knowledge Graphs: Build interactive DAG visualizations (directed acyclic graphs) on the dashboard displaying conceptual dependencies, topic hierarchies, and recommended prerequisite learning pathways directly extracted from ingested course materials. • Spaced Repetition & Exam Countdown Schedules: Combine the Bayesian Knowledge Tracing engine with the Ebbinghaus forgetting curve and SuperMemo SM-2 spaced repetition algorithms to generate dynamic daily revision schedules and review alerts tailored to an approaching exam date. • Multimodal Audio Briefs & Interactive Voice Tutoring: Implement real-time voice-driven tutor interactions and 2- minute synthesized audio briefs targeted specifically at a student's weakest topics for audio-first revision while commuting or walking. • Indian-Language & Mixed-Language Support (Hinglish/Regional): Support bilingual explanation modes (e.g., English textbooks and video lectures explained with conversational Hindi/Hinglish audio and text) to dramatically widen educational access and comprehension across diverse student cohorts. • Automatic Flashcard & Formula Sheet Generation: Enable one-click export of Anki flashcard decks, formula summary cheat sheets, and targeted mock exam PDFs based on concepts where student mastery falls below 60%.
Log in or sign up for Devpost to join the conversation.