OmniTutor: Multimodal AI Source-Grounded Learning System

Inspiration

As university students, we study across fundamentally fragmented learning materials: 200-page textbook PDFs, 60-slide PowerPoint decks, and hours of lecture video recordings. When students ask general-purpose AI models for help, two major problems occur:

Academic Hallucinations: Standard LLMs extrapolate convincing answers that do not exist anywhere in the instructor’s syllabus.
Missing Source Provenance: Students cannot quickly verify claims or jump to the exact diagram, slide, or video timestamp where the professor explained the concept.
Static Practice & Lack of Knowledge Tracing: Generic practice quizzes repeat memorized questions without tracking which concepts a student consistently misses.

We built OmniTutor to solve this: an end-to-end, zero-hallucination educational AI system that ingests multimodal course materials, provides strictly grounded answers with deep-linked citations, generates duplicate-free adaptive assessments, and tracks student mastery continuously.

What it does

OmniTutor operates across four interconnected pipelines:

Multimodal Content Ingestion (Pipeline A):
- Ingests .pdf notes, .pptx presentations, and .mp4 / .webm video recordings.
- Extracts slide hierarchies, lecture notes, and visual diagrams using PyMuPDF and python-pptx with OCR detection.
- Synchronizes video segments to exact MM:SS start/end timestamps.
- Preserves source provenance ({course_id, topic_id, page/slide/timestamp, chunk_id}) and indexes semantic vectors into ChromaDB.
Source-Grounded Tutoring with Zero-Hallucination Guardrails (Pipeline B):
- Retrieves verified evidence chunks using cosine similarity: $$\text{sim}(u, v) = \frac{u \cdot v}{|u|_2 |v|_2}$$
- Enforces a strict threshold guardrail ($\text{sim} \ge 0.35$). If a query is off-material (e.g. general trivia), the tutor strictly declines to guess: > "The uploaded course material does not establish the answer to this question."
- Returns grounded answers with clickable citation chips. Clicking a citation opens our synchronized document viewer, jumping directly to the cited PDF page, slide, or video timestamp.
Adaptive Assessment Engine (Pipeline C):
- Synthesizes scoped multiple-choice questions mapped to topic and difficulty (easy, medium, hard).
- Computes a SHA-256 hash of each question against attempt history to guarantee zero repeated questions.
- Provides instant grading and evidence-based explanations citing exact source chunks.
Continuous Learner Model & Knowledge Tracing (Pipeline D):
- Updates student topic mastery $M_t \in [0, 1]$ after each assessment using an Exponential Moving Average (EMA): $$M_t = (1 - \alpha) M_{t-1} + \alpha S_t \quad (\alpha = 0.35)$$ where $S_t$ is the normalized assessment score.
- Isolates repeatedly missed concepts into identified misconceptions.
- Dynamically recommends the next topic, difficulty level, and a tailored revision checklist.
Evaluation Suite:
- Built-in RAGAS & DeepEval-aligned tests verifying Faithfulness ($83\% - 94\%$) and Citation Precision ($100\%$).

How we built it

Backend: Built with FastAPI (Python 3.11) for high-throughput orchestration and structured REST endpoints.
Relational Storage: SQLAlchemy managing 6 core entities (courses, sources, topics, content_chunks, questions, student_progress).
Vector Database: ChromaDB with persistent ONNX embeddings (all-MiniLM-L6-v2) for semantic similarity retrieval.
Multimodal Processors:
- PyMuPDF for PDF rendering, page locator tracking, and diagram figure extraction.
- python-pptx for shape, title, and presenter note extraction.
- Whisper-aligned timestamp segmenter for lecture video playback.
Frontend: React 19 + TypeScript + Vite, styled with a custom dark-mode glassmorphic design system and Lucide icons.
Citation Navigation Modal: Custom synchronized viewer that deep-links to cited pages, slides, or seeks video playback.

Challenges we ran into

Cross-Modal Provenance Alignment: PDFs have page numbers, slide decks have slide numbers, and videos have millisecond timestamps. Unifying these into a single consistent citation model required designing a polymorphic locator schema that renders seamlessly in the browser.
Eliminating Subtle Hallucinations: Standard RAG pipelines often encourage the LLM to fill in gaps. We tuned a cosine distance rejection threshold that ensures answers only cite verified database records.
Question Repetition: Generating questions on the fly frequently produces near-duplicate phrasing. Implementing SHA-256 semantic hashing validated against student history solved this.

Accomplishments that we're proud of

True Multimodal Grounding: A student can click a citation chip in chat and immediately see the exact slide or jump to the exact video timestamp where the professor explained it.
Zero Hallucination Rejection: Cleanly refusing off-material questions rather than inventing facts.
Full-Stack Execution: Complete implementation across ingestion, vector storage, adaptive testing, and learner modeling with a responsive UI.
Automated Evaluation Metrics: Native integration of RAGAS and DeepEval principles directly in the application.

What we learned

How to design chunking pipelines that preserve document provenance across heterogeneous media formats.
How to bridge vector similarity search with relational databases to guarantee that citations refer to real source records.
How Bayesian Knowledge Tracing principles turn passive study sessions into an adaptive, personalized learning loop.

What's next for OmniTutor

In-Browser Audio Recording: Allowing students to record lectures directly and transcribe them live.
Socratic Voice Mode: Multi-turn voice dialogue guiding students to discover answers rather than simply displaying them.
LMS Integrations: One-click sync with Canvas, Blackboard, and Google Classroom.