Inspiration
Students often struggle to connect information spread across fragmented study materials like lecture recordings, slides, and heavy textbooksspan_3span_3. Traditional learning tools don't offer dynamic feedback or verifiable sources, leading to inefficient studying and dynamic knowledge gaps. We built OmniTutor AI to bridge this gap by converting multi-format course content into a unified, interactive, and source-cited learning assistantspan_4span_4.
What it does
OmniTutor AI is an intelligent study companion designed for adaptive learningspan_5span_5:
- Multi-Modal Ingestion: Unifies lecture videos, slide decks, and textbooks into a single, cohesive vector-based knowledge graphspan_6span_6.
- Source-Cited Q&A: Delivers accurate answers to complex student questions, providing precise citations back to specific video timestamps, slide numbers, or textbook paragraphsspan_7span_7.
- Adaptive Assessments: Automatically generates diagnostic quizzes and real-time practice problems tailored to the user's current mastery levelspan_8span_8.
- Personalized Feedback: Analyzes quiz performance to identify weak areas and dynamically curates targeted revision paths.
How we built it
- Data Processing & Pipeline: Extracted text from textbooks and slides using OCR and PDF parsing utilities, while extracting transcriptions and timestamps from lecture videos.
- Knowledge Retrieval (RAG): Segmented and indexed data into a high-performance vector database using embedding models to enable accurate semantic retrieval.
- LLM Engine: Integrated a multimodal LLM framework tuned to prioritize source attribution and prevent hallucinationsspan_9span_9.
- User Interface: Built a responsive interactive dashboard using React and Tailwind CSS for seamless document exploration and real-time chat interactions.
Challenges we faced
- Cross-Modal Alignment: Synchronizing video timestamps with static text concepts to ensure citations pointed directly to the exact segment in video lecturesspan_10span_10.
- Context Preservation: Chunking long textbooks and detailed slide decks without losing cross-page context or math equation formatting during retrieval.
- Latency Optimization: Streamlining embedding lookups and LLM streaming responses to maintain a fast, conversational feedback loop.
Accomplishments that we're proud of
- Achieving millimeter-precise source attribution across video transcripts and PDF documentsspan_11span_11.
- Creating an adaptive assessment system that dynamically recalibrates difficulty based on real-time student performancespan_12span_12.
- Delivering an intuitive UI that unifies diverse media streams into one cohesive workspacespan_13span_13.
What we learned
- Effective prompt engineering and guardrails are crucial when requiring strictly cited outputs from LLM pipelines.
- Multimodal Retrieval-Augmented Generation (RAG) requires careful chunking strategies when combining structured slides and unstructured video scripts.
What's next for OmniTutor AI
- Expanding support for live voice-based tutoring and real-time interactive audio discussions.
- Adding collaborative study rooms so peers can learn and test each other using shared knowledge bases.
Built With
- fastapi
- langchain
- openai
- pinecone
- pypdf2
- python
- react
- tailwind-css
- typescript
Log in or sign up for Devpost to join the conversation.