OmniTutor AI

Adaptive multimodal learning engine uniting video lectures, slide decks, and textbooks into a source-cited interactive tutor.

Comment

Inspiration

Students navigate fragmented learning resources: 60-minute video lectures on YouTube, dense 400-page textbook PDFs, and high-level lecture slides. Traditional conversational AI study assistants frequently fail in high-stakes technical domains due to hallucinated definitions and an absence of temporal and spatial provenance. When a model provides an ungrounded explanation, students cannot verify which paragraph of the syllabus or which segment of the lecture confirms it.

OmniTutor AI bridges this gap. It acts as an adaptive, multimodal study companion that unifies unstructured video lectures, slide decks, and textbooks into a deterministic, source-cited knowledge base with dynamic diagnostic assessments.

⚙️ What It Does

OmniTutor ingests multiple educational modalities simultaneously and exposes two core workflows:

  1. Multimodal Grounded Retrieval & Citation:

    • Ingests slide decks and textbooks via deterministic page-level text and layout extraction.
    • Ingests video lectures by extracting synchronized timestamped transcripts.
    • Synthesizes cross-modal responses where every factual premise is explicitly anchored to physical evidence (e.g., [Slide/PDF Pg. 14] or [Lecture Video 12:45]).

  2. Adaptive Diagnostic Assessment Engine:

    • Instead of static quizzes, the system synthesizes dynamic diagnostic questions targeting core conceptual dependencies from the ingested materials.
    • If a learner provides an incorrect response, the engine isolates the knowledge gap and delivers adaptive scaffolding hints pointing directly back to the exact video timestamp or slide page.

🛠️ How We Built It

OmniTutor was built with a lightweight, high-throughput Python architecture:

  • Multimodal Extraction Layer:
    • Document Ingestion: PyPDF2 parses page-delimited tokens and document structure.
    • Temporal Video Stream: youtube-transcript-api extracts fine-grained millisecond timestamps and subtitle chunks directly from lecture videos.
  • Reasoning & Citation Engine:
    • Built using Google Gemini 2.5 Flash (google-genai SDK) configured with strict system prompt schemas enforcing verifiable in-text source tagging.
  • Interactive Frontend:
    • Developed using Streamlit, organizing the ingestion pipeline, source-grounded chat interface, and adaptive quiz modules into responsive split tabs.

🚧 Challenges We Faced

  • Temporal vs. Spatial Alignment: PDFs index knowledge by discrete pages, whereas video lectures structure information across a continuous time continuum. Mapping queries to cross-modal references required standardizing citation metadata into a unified prompt context.
  • Mitigating Hallucinations in Educational QA: Standard LLM completions tend to add extraneous explanations. Enforcing strict source boundaries required schema tuning to ensure that if a topic is not present in the ingested slide or video chunk, the system declines to speculate.

🧠 What We Learned

  • How to implement deterministic prompt constraints for academic provenance.
  • Designing lightweight, low-latency multimodal pipelines without incurring heavy vector database infrastructure overhead for on-the-fly student sessions.

🚀 What's Next for OmniTutor AI

  • Direct visual frame indexing using Vision-Language Models (VLMs) to parse mathematical equations and circuit diagrams directly from video keyframes.
  • Integration of spaced-repetition schedules (SM-2 algorithm) tied to student error logs.
  • Localized edge deployment for low-bandwidth rural educational environments.

Built With

Share this project:

Updates

Submission history