Inspiration
Students often study using lecture slides, textbooks, mathematical equations, diagrams, and recorded lectures. However, understanding these materials, finding reliable explanations, and identifying weak topics can be time-consuming. I wanted to build a learning assistant that connects course materials with personalized tutoring and practice, rather than providing generic AI answers.
What it does
StudyPilot AI is a local-first, multimodal AI tutoring prototype built for the Multimodal AI Hackathon 2026, Track D. It processes uploaded course PDFs, supports optional lecture-video frame analysis, retrieves relevant learning material, generates source-grounded explanations and practice questions, and tracks topic-level mastery to recommend revision.
How we built it
I built StudyPilot using Python and Streamlit, with PyMuPDF for PDF processing and Ollama for locally hosted language and vision models. The application organizes extracted content into chunks, retrieves relevant passages for tutoring and assessment, and maintains a learner model that updates from assessment results and feedback. It also includes a course map, source inspection, and a prototype evaluation dashboard. FFmpeg supports optional video-frame extraction.
Challenges we ran into
One of the biggest challenges was extracting meaningful educational content from PDFs containing equations, diagrams, and complex layouts. Vision-based recovery can improve extraction but increases processing time. Generating multiple valid assessment questions, preserving mathematical notation, handling model responses reliably, and keeping the interface responsive during long local inference tasks were also challenging.
Accomplishments that we're proud of
I brought several parts of the learning workflow into one application: course-material ingestion, source-grounded tutoring, adaptive practice, learner progress tracking, and evaluation. I'm particularly proud of the local-first approach and the ability to connect generated learning activities with source passages rather than relying entirely on generic responses.
What we learned
Building StudyPilot taught me how retrieval-augmented generation connects source documents to language-model responses, how multimodal extraction can help with visual educational content, and how assessment results can inform a learner model. I also learned that reliable AI applications require careful handling of extraction quality, model latency, generated output validation, and end-to-end testing.
What's next for StudyPilot
Next, I want to improve mathematical and diagram extraction, strengthen assessment verification, evaluate tutoring quality on a fixed test set, and make learner recommendations more reliable. I also want to simplify deployment so students can access StudyPilot more easily without manually configuring local models.
Built With
- ai
- artificial
- edtech
- education
- generative
- intelligence
- language
- learning
- machine
- multimodal
- natural
- ollama
- personalized
- python
- rag
- streamlit
- tutor
