OmniTutor AI

Inspiration

As a computer science engineering student balancing complex coursework, I know firsthand the friction of preparing for an exam when material is scattered across two-hour lecture videos, massive textbooks, and dozens of slide decks. OmniTutor AI was born from the need to eliminate this chaos. Instead of helplessly scrubbing through video timelines or hitting Ctrl+F on massive PDFs, I wanted to build a unified system that acts as a true study companion—one that actually teaches rather than just spoon-feeding answers.

What it does

OmniTutor AI ingests diverse course materials (video transcripts, slide decks, and textbooks) and unifies them into a single, source-cited knowledge base. When a student asks a question or requests a quiz, the system uses a Retrieval-Augmented Generation (RAG) architecture to find the most relevant context and employs a Socratic teaching style to guide the student to the answer, always citing exactly where the information came from (e.g., "As seen in Lecture 3, Slide 12").

How we built it

The core backend is built with Python, orchestrating the data ingestion and retrieval pipelines. We used OCR and parsing libraries to extract text from textbooks and slides, and an audio transcription model to generate timestamped text from lecture videos. These documents are split into semantic chunks and converted into high-dimensional vectors stored in ChromaDB. When a user queries the system, we compute the cosine similarity between the query vector $q$ and the document chunk vectors $d_i$ to retrieve the most relevant context:$$\text{similarity}(q, d_i) = \frac{q \cdot d_i}{\Vert{}q\Vert{} \Vert{}d_i\Vert{}}$$The retrieved chunks are passed as context to the Google Gemini API. We utilized Gemini for its robust multimodal reasoning, and the entire interactive dashboard is served via Streamlit.

Challenges we ran into

The toughest hurdle was prompt engineering for the Socratic method. Generative models naturally want to be overly helpful by providing the direct answer immediately. Forcing the LLM to hold back the answer, evaluate the student's current understanding, and ask a guiding question required extensive prompt iteration. Additionally, aligning multimodal metadata was tricky. Ensuring that a retrieved text chunk from a video transcript accurately carried its timestamp metadata all the way through the generation phase—so the bot could output clickable video timestamps—required very strict data structures during the initial chunking phase.

Accomplishments that we're proud of

I am incredibly proud of the strict source-citation engine. The AI rarely hallucinates because it is heavily constrained by the retrieved context, and it builds immediate trust with the user by pointing directly to the exact page or video minute that validates its explanation

What we learned

Transitioning concepts from recent workshops on Generative AI and advanced technical coursework into a fully functional, deployed architecture was a massive leap. Building this project provided critical hands-on experience with vector math in practical applications, handling multimodal context windows, and refining API latency for a smoother user experience

What's next for OmniTutor AI

The immediate next step is expanding the adaptive assessment module. I plan to implement a spaced-repetition algorithm matrix that tracks a user's proficiency across specific topics over time. If the system detects a recurring weakness in a concept, it will dynamically adjust the difficulty of future generated quizzes and automatically surface remedial reading materials from the knowledge base.

Built With

• artificial-intelligence
• chromadb
• edtech
• gemini-api
• generative-ai
• google-gemini
• langchain
• llamaindex
• llm
• machine-learning
• multimodal
• opencv
• pymupdf
• python
• rag
• sentence-transformers
• sqlite
• streamlit
• tesseract-ocr
• whisper