StudyPilot

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

Inspiration

In today’s digital classroom, learning materials are scattered across a chaotic mix of formats: two-hour lecture videos, dense textbook PDFs, and dozens of slide decks. When studying for exams, students spend more time searching for information than actually learning it. Furthermore, traditional studying is a "one-size-fits-all" process that completely ignores individual memory retention rates.

We were inspired to build StudyPilot to solve this. We wanted to create a highly personalized, multimodal AI tutor that not only centralizes and understands all your course formats, but actively adapts to your cognitive state, teaching you exactly what you need to know, exactly when you are about to forget it.

What it does

StudyPilot is a highly personalized AI study companion that unifies your entire curriculum into one interactive platform and adapts its teaching style to your unique learning pace.

Instead of juggling multiple tabs and formats, here is how StudyPilot handles the heavy lifting:

Unifies Multimodal Materials: You can upload all your course materials in one place—whether they are dense PDF textbooks, PowerPoint slide decks, or MP4 lecture videos. StudyPilot digests them all, transcribing audio and extracting visual text, to build a comprehensive, searchable knowledge base. Source-Grounded Tutoring: It features an interactive AI Tutor Chat that answers your specific questions. Instead of guessing or hallucinating, it strictly grounds its answers in your uploaded materials. Every response includes precise citations (e.g., jumping you directly to a specific video timestamp or slide number) so you can verify the context yourself. Adaptive Assessments: StudyPilot automatically generates dynamic quizzes to test your knowledge. As you answer questions, it uses a cognitive algorithm called Bayesian Knowledge Tracing (BKT) to accurately track your mastery of underlying concepts in real-time. Optimized Study Plans: Using the Ebbinghaus Forgetting Curve, the dashboard actually predicts when your memory of a specific topic is about to fade. It automatically generates a prioritized study schedule for you, highlighting urgent topics that need spaced-repetition review, alongside new concepts you are ready to tackle.

How we built it

We architected StudyPilot using a modern, scalable tech stack, split into a robust Python backend for handling the heavy AI workloads and a highly interactive React frontend for the user experience.

1. Multimodal AI Ingestion Pipeline The core of our backend is built with Python and FastAPI. To handle the ingestion of diverse course materials, we built a custom multimodal parsing pipeline:

Audio/Video: We utilized ffmpeg combined with OpenAI's Whisper model to transcribe MP4 lecture videos, ensuring we mapped the generated text precisely to video timestamps. Visuals & Text: We leveraged Google Gemini Pro and Gemini Pro Vision to extract text, interpret diagrams from PDF textbooks, and pull context from PowerPoint (.pptx) slide decks.

1. Vector Database & Source Grounding Once the multimodal data is parsed and intelligently chunked, we generate embeddings using GoogleGenerativeAIEmbeddings.

These embeddings are stored in a ChromaDB / Pinecone vector database (orchestrated via LangChain). When a student asks the AI Tutor a question, our backend runs a semantic similarity search to retrieve the most relevant chunks. We pass this context to the LLM to generate a strictly grounded response, ensuring the AI cites the exact slide number, PDF page, or video timestamp rather than hallucinating answers.

1. The Cognitive Learner Model To make the platform truly adaptive, we implemented educational psychology algorithms from scratch in Python:

Bayesian Knowledge Tracing (BKT): We probabilisticly model student mastery based on their quiz performance. We tune parameters for Slip (knowing the concept but getting the question wrong) and Guess (not knowing it but getting it right) to accurately update the student's mastery state after every interaction. Spaced Repetition: We paired the BKT model with the Ebbinghaus Forgetting Curve to calculate memory decay over time, allowing our system to trigger review sessions exactly when a student is mathematically predicted to forget a concept.

1. The Frontend Experience The user interface was built using React and Vite, styled entirely with Tailwind CSS. We focused heavily on creating a premium, modern aesthetic:

Interactive Visualizations: We used Recharts to plot complex data—like overlaying the student's actual BKT mastery against their predicted forgetting curve, and rendering radar charts to show domain competency. UI/UX: We incorporated a "glassmorphism" design system and utilized Framer Motion for fluid page transitions, staggered list animations, and interactive hover states to make the dashboard feel alive and highly engaging.

Challenges we ran into

Building a platform that sits at the intersection of generative AI, multimodal data processing, and educational psychology presented several tough technical hurdles:

Multimodal Synchronization: One of our biggest challenges was processing MP4 lecture videos. Transcribing the audio was straightforward, but intelligently chunking that text and syncing it precisely with the correct visual video frames proved difficult. We had to implement a custom parser to ensure that when the AI cited a video, clicking the citation dropped the student at the exact correct second of the lecture. Tuning the Cognitive Learner Model: Implementing the Bayesian Knowledge Tracing (BKT) algorithm required us to set initial probabilities for Slip (knowing a concept but getting a question wrong by mistake) and Guess (not knowing a concept but guessing the quiz answer correctly). If we tuned these incorrectly, the platform would either constantly harass the user with reviews they didn't need, or skip over topics they hadn't actually mastered. It took dozens of simulated student evaluations to balance these hyperparameters. Taming LLM Hallucinations: We wanted the AI Tutor to act as a strict teaching assistant for a specific course, not a general-purpose chatbot. Ensuring the LLM answered strictly using the retrieved context from the vector database—and politely declining to answer out-of-scope questions—required extensive prompt engineering and fine-tuning our vector similarity search thresholds. Complex Frontend Visualizations: Translating complex mathematical data (like overlapping decay curves and real-time mastery scores) into a beautiful, intuitive UI was challenging. Getting the Recharts data structures to play nicely with Framer Motion's stagger animations required careful React state management to ensure the dashboard felt snappy and responsive rather than overwhelming.

Accomplishments that we're proud of

Beyond Basic Chatbots: We are incredibly proud that we didn't just build another "PDF chatbot." By successfully implementing and tuning the Bayesian Knowledge Tracing (BKT) and Ebbinghaus Forgetting Curve algorithms from scratch, we built a system with actual cognitive science backing that actively models the user's brain state. True Source Grounding: Achieving accurate source grounding across multiple modalities was a massive win. We are proud that our AI Tutor doesn't just give answers, but acts as a hyper-accurate librarian—pointing the user to the exact slide or the precise timestamp in a video where a concept was taught. A Premium User Experience: We are thrilled with how the frontend turned out. Translating complex AI data into a visually stunning, glassmorphic UI makes the platform feel like a production-ready product rather than a hackathon prototype. The interactive charts plotting actual mastery against predicted memory decay are both technically complex and highly intuitive. The Full-Stack Integration: Successfully stitching together a modern React frontend, a FastAPI backend, vector databases, and multiple generative AI models (Gemini Pro, Vision, Whisper) into a seamless, low-latency loop is a major engineering accomplishment for our team over the span of a single hackathon.

What we learned

Building StudyPilot was an incredible masterclass in bridging the gap between cutting-edge AI and traditional educational psychology. Here are our biggest takeaways:

Multimodal RAG is More Than Just Text: We learned that building a Retrieval-Augmented Generation pipeline for videos and slides is vastly different than parsing basic text documents. We had to learn how to intelligently chunk audio transcripts by timestamp and extract spatial metadata from images so the LLM could "see" the context properly. The Power of Educational Algorithms: We discovered that simply wrapping an LLM interface around data isn't enough to create a tutor. Learning how to implement Bayesian Knowledge Tracing taught us how to probabilistically model a user's hidden "mastery state" over time, transforming a static chatbot into an active, personalized educator. Prompt Engineering for Strict Grounding: We learned how difficult it can be to keep generative AI strictly within the bounds of a provided context. We deeply leveled up our prompt engineering skills to enforce strict citation rules and prevent hallucinated answers. Advanced Frontend State Management: Rendering dynamic, overlapping data curves in React using Recharts while simultaneously managing Framer Motion transitions taught us a lot about advanced component lifecycle and state management. We learned how to prioritize rendering performance so the dashboard remained buttery smooth despite heavy data calculations.

What's next for StudyPilot

While we are incredibly proud of what we built over the hackathon, we view this as just the foundational layer for the future of personalized education. Here is what we plan to build next:

Real-Time Lecture Integration: We plan to integrate live-streaming transcription capabilities. Imagine sitting in a lecture with StudyPilot running—it actively listens, summarizes the material, and injects the live context straight into your Learner Model in real-time. Deep Knowledge Tracing (DKT): Currently, our platform uses Bayesian logic. We want to upgrade the core cognitive engine to use Deep Knowledge Tracing (using Recurrent Neural Networks) to predict student mastery and slipping rates with even higher accuracy across larger datasets. GraphRAG Implementation: We want to transition from a pure semantic vector search to a Knowledge Graph structure. This would allow the AI to actively map how concepts from a video relate to a diagram in chapter 4 of the textbook, helping students visualize the "big picture" connections in their coursework. Collaborative Peer Matching: We plan to introduce a social feature that analyzes the mastery profiles of different students in a cohort and pairs them together based on complementary strengths—for example, matching a student struggling with backpropagation with a peer who has a 95% mastery score in that exact topic.

Built With

• framer-motion
• google-cloud
• javascriptfrontend-frameworks-&-ui:-react
• langchain
• languages:-python
• pandas
• pinecone
• ragas
• rechartsbackend-&-apis:-fastapi
• rest-apiai-&-machine-learning:-gemini
• tailwindcss
• typescript
• uvicorn
• vector-databasedata-processing-&-evaluation:-ffmpeg
• vite
• whisperdatabases-&-storage:-chromadb