Inspiration

Students today learn from many places at once: recorded lectures, textbook PDFs, and slide decks. But these sources never talk to each other, and no tool tells a student what they actually understand and what they are only pretending to understand. Generic chatbots make it worse, because they answer confidently without telling you where the answer came from. We wanted a tutor that only teaches from the student's own material, shows its sources, and adapts to what the student really knows.

What it does

StudyCompanion AI turns lecture videos, textbooks (PDF), and slide decks (PPTX) into one source-cited knowledge base, then uses it to tutor and assess the student.

• Upload: drag and drop PDF, PPTX, or video files, or paste a YouTube URL. Live progress streams to the screen while the file is processed.
• Tutor Chat: answers questions grounded in the uploaded material with inline citations like [1][2]. If the question isn't covered by the sources, a relevance gate makes it refuse instead of guessing.
• Knowledge Map: automatically extracts topics and their prerequisites into an interactive graph.
• Adaptive Assessment: generates MCQ, short-answer, and numerical questions that are verified, deduplicated, and tagged by topic.
• Dashboard: shows a mastery heatmap, a spaced-repetition schedule, and assessment history.

A Bayesian Knowledge Tracing (BKT) learner model updates mastery after every answer, so the questions and review schedule adapt to each student.

How we built it

• Frontend: Next.js 14 with React Flow for the knowledge graph.
• Backend: FastAPI with SQLAlchemy. SQLite for local development, and the same ORM works with PostgreSQL in production.
• Multimodal ingestion: Whisper for speech-to-text on videos, and Gemini 1.5 Flash as the vision-language model to caption slides and figures.
• Retrieval: ChromaDB for vectors, with hybrid BM25 + dense search combined by Reciprocal Rank Fusion, followed by a cross-encoder reranker for the final top results.
• Generation: Gemini 1.5 Flash for grounded answers, quiz generation, and topic extraction.
• Learner model: BKT, chosen over IRT because it is interpretable and works with few observations.
• Evaluation: a RAGAS-inspired pipeline measuring faithfulness, relevancy, context precision, and refusal behavior, plus an adaptive vs. random baseline comparison across 3 simulated student profiles.
• Deployment: Docker Compose for one-command setup.

Challenges we ran into

• Keeping answers faithful to the sources and making the system refuse off-topic questions instead of hallucinating.
• Combining very different inputs (video transcripts, PDF text, slide images) into one consistent, searchable format with accurate source locations for citations.
• Dense equations in slides are hard for vision-language models to caption accurately.
• BKT needs good slip, guess, and transit parameters, but with limited data we had to start from sensible defaults.
• Generating quiz questions that are correct and not repetitive, which needed a verification and deduplication step.

Accomplishments that we're proud of

• A full pipeline from raw lecture video to a cited answer, adaptive quiz, and mastery dashboard.
• Hybrid retrieval with reranking, so exact keyword matches and semantic matches both work.
• A built-in evaluation harness that measures the system and shows adaptive selection compared with a random baseline, instead of only claiming it works.
• Honest documentation of limitations in the README.
• (Add your actual eval numbers here, e.g. faithfulness and relevancy scores.)

What we learned

• Retrieval quality matters more than model size: hybrid search plus reranking improved answers more than anything we changed in the prompt.
• Refusing to answer is a feature. Students trust a tutor more when it says "this isn't in your material."
• Simple, interpretable models like BKT are easier to debug, explain, and evaluate than heavier alternatives.
• Evaluating early shows weaknesses that a demo hides.

What's next for Personalized Tutoring & Adaptive Learning

• Estimate BKT parameters per student from real usage data instead of defaults.
• Use stronger vision models for equation-heavy STEM content, and add a proper NLI-based faithfulness metric.
• Support more sources and languages, and handle noisy audio and accents better.
• Move to PostgreSQL and a hosted vector store for classroom-scale use.
• Pilot it with real students and compare learning outcomes against traditional study.

Built With

• antigravity