📚 Building Learning Companion: The Full Story

💡 What Inspired Us

Traditional learning tools often treat study materials as static silos, leaving students to manually sift through dense textbooks, scattered lecture slides, and lengthy video transcripts without a clear path to mastery. We were inspired to build Learning Companion to bridge the gap between passive content consumption and active, structured learning. Our goal was to create a source-grounded, intelligent study app that unifies diverse multimedia formats into a single, reliable knowledge base—ensuring every explanation is strictly backed by citations while dynamically adapting to a student's evolving understanding through Bayesian Knowledge Tracing.

🏗️ How We Built the Project

The project follows a strict architectural blueprint separating a FastAPI backend and a React + Vite frontend, coordinated via LangChain and LangGraph:

1. Multimodal Ingestion (G0): We built ingestion pipelines capable of processing PDFs, PowerPoint presentations (.ppt, .pptx), text files, subtitle transcripts (.srt, .vtt), and images (.jpg, .png) into structured CitationUnit chunks.
2. Grounded Orchestration (LangGraph): Rather than raw SDK calls, multi-step RAG workflows—such as retrieval, grading, expansion, generation, and verification—are managed inside typed LangGraph state machines (thread_id = student_id).
3. Vector Retrieval & Knowledge Graph: Retrieved chunks are stored in Chroma (using sentence-transformers/all-MiniLM-L6-v2 embeddings) combined with keyword re-ranking and a NetworkX concept graph.
4. Adaptive Learning Loop (BKT): We implemented a Bayesian Knowledge Tracing engine coupled with a daily forgetting curve ($\text{Decay} = 0.995 / \text{day}$) to dynamically adjust question difficulty and manage student revision queues.

5. Interfaces: The backend exposes clean REST endpoints (/ingest, /chat, /quiz, /mastery), complemented by a Streamlit debugging interface and an asymmetric editorial React + Tailwind frontend.

🧠 What We Learned

• Stateful Graph Control: Managing conversation and retrieval states through LangGraph checkpointers vastly improved the reliability of complex generation loops compared to traditional linear scripts.
• Strict Source Attribution: Designing metadata schemas that track exact page numbers, slide indices, and subtitle time ranges ($\text{t_start} / \text{t_end}$) is crucial for preventing hallucinations and maintaining trust.
• Hybrid Retrieval Balance: Combining dense vector similarity with sparse keyword re-ranking prevents niche technical terminology from getting lost during open-ended tutor queries.

⚠️ Challenges We Faced

• Unifying Heterogeneous Modalities: Harmonizing text chunks, slide decks, and video transcripts into a single consistent citation format required robust error handling and specialized loaders.
• Rate Limits and Offline Resilience: Depending on free-tier LLM providers introduced rate-limiting hurdles. We overcame this by wrapping LLM calls in fallback heuristics, ensuring the application remains fully testable offline via smoke tests.
• Tuning Adaptive Parameters: Calibrating BKT priors (P_init = 0.30, Learn = 0.25, Slip = 0.10, Guess = 0.20) required fine-tuning to ensure mastery progression accurately matched natural student learning curves without being overly penalizing.

Built With

• artificial-intelligence
• chromadb
• docker
• edtech
• fastapi
• git
• github
• langchain
• langgraph
• machine-learning
• multimodal
• networkx
• openrouter
• pydantic
• python
• rag
• ragas
• react
• rest-api
• streamlit
• sympy
• tailwind-css
• typescript
• vector-database
• vite