OmniTutor: Multimodal AI Study Companion

MAIN PAGE..
ACCESS PAGE..
MEDIA AND MODULES..
QUIZ SESSION..
STUDENT KNOWLEDGE GRAPH PAGE..

Inspiration As 3rd-year Electrical and Electronics Engineering students at Bannari Amman Institute of Technology, we face a recurring struggle before every semester exam: The Fragmented Study Paradox.

When studying complex technical subjects like Power System Analysis or Power Electronics, our learning materials are scattered across multiple platforms. A crucial 2-minute formula derivation is buried somewhere in a 90-minute lecture recording, while presentation slides, textbooks, and university question banks sit in disconnected folders.

When students consult standard AI chatbots, they encounter generic, unverified explanations that frequently hallucinate and fail to match the specific syllabus or our professor's exam requirements. We built OmniTutor to bridge this gap: an autonomous study companion that ingests lecture videos, audio, slides, and question bank PDFs to deliver zero-hallucination answers backed by exact video timestamp citations and personalized adaptive quizzes.

What It Does OmniTutor transforms static course files into an interactive, grounded academic tutor:

Unified Multimodal Ingestion: Ingests lecture videos (.mp4), audio recordings (.mp3/.wav), and course question bank PDFs (.pdf) into a single unified session.
Source-Cited Q&A: Answers conceptual and numerical questions with exact clickable video timestamps (e.g., [Video: 10:15 - 12:40]) and document page numbers (e.g., [Page: 4]), eliminating hallucinations.
Mathematical Precision with LaTeX: Solves numerical problems step-by-step with real governing formulas, such as per-unit base transformations and voltage regulation: $$\%VR = \frac{V_{NL} - V_{FL}}{V_{FL}} \times 100\% \approx \frac{I(R\cos\phi \pm X\sin\phi)}{V_2} \times 100\%$$ $$Z_{pu(new)} = Z_{pu(old)} \times \left(\frac{V_{base(old)}}{V_{base(new)}}\right)^2 \times \left(\frac{S_{base(new)}}{S_{base(old)}}\right)$$
Agentic Adaptive Assessment: Dynamically generates multiple-choice quizzes that scale in difficulty (Easy, Medium, Hard) based on student performance.
Knowledge Gap Diagnosis: Evaluates student quiz answers, detects root-cause misconceptions, and formulates a 3-step remedial review plan pointing directly to the relevant video segment and page.
360° Competency Radar Profile: Visualizes module-wise proficiency and competency radar graphs to help students target high-priority revision topics before exams.

How We Built It

Multimodal AI Engine: Powered by Google Gemini 1.5 Flash utilizing its native 1M+ token multimodal context window for video frame understanding and semantic document grounding, backed by an auto-model fallback routine.
Backend Architecture: Built with Python 3.14, FastAPI REST endpoints, PyPDF for local text extraction, and Uvicorn for asynchronous server execution.
Frontend Layer: Designed two complementary interfaces:
- A modern, responsive Glassmorphism Web Dashboard built with HTML5, Tailwind CSS, Lucide Icons, and Chart.js for real-time chat and radar visualizations.
- A dedicated Streamlit Interactive Dashboard with embedded video playback and Plotly charts.
Reliability & Local Fallback: Engineered a local multi-document semantic search engine so the application remains completely functional with zero downtime even in offline or API-limited environments.

🧗 Challenges We Ran Into

Multimodal Temporal Grounding: Aligning timestamps across long video streams with static pages in question bank PDFs without introducing hallucinated references.
Dynamic Multi-Document Filtering: Preventing cross-topic confusion when students upload multiple distinct semester files (e.g., 22EE501 Power Systems alongside 22EE502 Power Electronics and 22IT701). We solved this by developing a weighted semantic relevance search that prioritizes course codes and specific unit modules.
Designing True Agentic Feedback: Moving beyond traditional grading percentages to build an evaluation agent that diagnoses the exact mathematical or conceptual reason a student missed a question.

Accomplishments That We're Proud Of

Tested and verified live with real engineering question banks (22EE501, 22EE502, 22IT701), accurately retrieving analytical formulas, problem excerpts, and page citations.
Built and deployed a production-grade full-stack web application featuring zero-hallucination citations, responsive chat, dynamic quiz generation, and competency radar analytics.
Completed full end-to-end integration matching all requirements of Track D: Personalized Tutoring & Adaptive Learning.

What We Learned

Advanced Multimodal RAG architectures, prompt orchestrations, and long-context processing with Google Gemini 1.5.
Asynchronous API design and modular architecture using FastAPI and Uvicorn.
Interactive educational data visualization using Chart.js radar plots and modern Tailwind CSS UI paradigms.
Team collaboration, time-boxed hackathon delivery, and project packaging for national-level competitions.

What's Next for OmniTutor

Speech-to-Speech Voice Mode: Adding real-time voice interaction for hands-free problem-solving during calculations.
University LMS Integration: Developing plug-and-play extensions for Moodle, Canvas, and Google Classroom so educators can inspect class-wide knowledge gaps.
Predictive Exam Readiness Index: Developing machine learning models to forecast student exam percentiles and suggest optimized revision timetables.