Inspiration## Inspiration
Higher education students face a fragmented learning environment: course material is scattered across lecture videos, slide decks, and PDF textbooks. Generic LLMs frequently produce ungrounded answers or hallucinations decoupled from official course syllabi. We built AegisAI to create a trustworthy, source-grounded study companion that unifies multimodal materials and dynamically adapts to each student's evolving mastery state.
What It Does
AegisAI is a complete multimodal tutoring and adaptive assessment system:
- Multimodal Ingestion Engine: Automated extraction from PDF textbooks (PyMuPDF), slide decks (visual diagrams via Gemini Vision), and lecture recordings (Whisper timestamps).
- Source-Grounded Tutor: Chat engine offering inline citations to exact slide numbers, textbook pages, and clickable video timestamps, with strict refusal protocols for out-of-corpus queries.
- Adaptive Assessment & Learner Model: Generates non-repetitive quizzes calibrated by 2PL IRT difficulty and tracks per-topic skill mastery using Bayesian Knowledge Tracing (BKT).
- Interactive Knowledge DAG: Visual node graph displaying topic prerequisites and real-time student mastery state.
How We Built It
- Frontend: Next.js 14, React Flow (interactive DAG visualization), Tailwind CSS.
- Backend & Vector Database: FastAPI, Qdrant (hybrid vector/lexical search), NetworkX (concept dependency graph).
- Inference & Vision: Gemini 2.5 Flash for vision extraction/diagram parsing and Whisper-timestamped for video temporal synchronization.
- Learner Model: Bayesian Knowledge Tracing update equations: $$P(L_t \mid o_t = 1) = \frac{P(L_t)(1 - P(S))}{P(L_t)(1 - P(S)) + (1 - P(L_t))P(G)}$$
- System Evaluation: Evaluated using RAGAS and DeepEval frameworks for Faithfulness, Answer Relevancy, and Context Precision.
Challenges We Faced
- Cross-Modal Alignment: Synchronizing temporal audio timestamps with static slide diagrams without manual transcription annotations.
- Preventing Question Hallucinations: Building a secondary cross-model verification pipeline that solves generated questions independently to filter out inaccurate answer keys.
- Out-of-Corpus Refusal Logic: Calibrating strict cosine similarity thresholds ($\tau = 0.55$) to refuse unsupported queries without degrading standard contextual answers.
What We Learned & Future Roadmap
- Learned how to implement quantitative IRT item parameter calibration alongside RAG architecture.
- Future work includes integrating Web Audio live voice tutoring sessions and expanded multilingual Hindi/Hinglish explanation support.
Made with Google studio
Log in or sign up for Devpost to join the conversation.