Groundwork · Multimodal AI Study Companion

Source-grounded multimodal AI tutor with exact PDF quadrant bounding boxes, exact-moment YouTube citations, zero-hallucination sandboxed math verification, and sealed Bayesian Knowledge Tracing.

Comment

Inspiration

General AI chatbots often hallucinate, lose context, or invent external facts when students ask questions about course lectures, slides, and textbooks. For Track D (Personalized Tutoring & Adaptive Learning), we were inspired to build Groundwork: an end-to-end, source-grounded multimodal study companion that formally models the learner while maintaining strict provenance across heterogeneous course materials.

What it does

Groundwork normalizes and unifies lecture videos, slide decks, recitations, and textbooks into a searchable knowledge base without manual preprocessing[cite: 25, 27].

  • Exact Multimodal Grounding: It tutors students using only retrieved syllabus materials. Citations open PDF slide handouts with exact quadrant bounding boxes, or open YouTube lectures within 9.9 seconds of the concept being spoken[cite: 25, 27].
  • Strict Boundary Enforcement: Queries outside the syllabus (e.g., "What is the capital of France?") are refused or isolated strictly inside a dedicated "Beyond your course" container[cite: 25, 27].
  • Zero Arithmetic Hallucination: Numerical problems are never graded by LLM estimates. Student answers are verified against an AST-sandboxed Python subprocess using exact rational arithmetic (fractions.Fraction)[cite: 25, 27].
  • Principled Learner Model: A sealed Bayesian Knowledge Tracing (BKT) engine tracks per-topic mastery, driving an adaptive "Study Next" algorithm that topologically prioritizes foundational axioms before advanced concepts[cite: 25, 27].

How we built it

  • Backend: FastAPI (Python 3.12) running on Render Free Tier, backed by Firebase Authentication and the Firestore Admin SDK[cite: 25, 26, 31].
  • Frontend: Next.js 16 (App Router, TypeScript) styled with Tailwind CSS and Base UI, hosted on Vercel Hobby[cite: 26, 27, 31].
  • AI & Processing: We orchestrate a free-tier Google Gemini model chain (gemini-3.8-flash $\rightarrow$ gemini-3.5-flash $\rightarrow$ gemini-3.1-flash-lite)[cite: 25, 26, 31]. Embeddings are generated locally using BAAI/bge-small-en-v1.5, and video audio is transcribed locally via Faster-Whisper medium.en to ensure zero API quota waste[cite: 25, 26, 31].
  • Reproducibility: We packaged the entire authenticated stack (Firebase emulators, backend API, Next.js standalone runner, and pre-warmed cache) into a one-command docker compose up execution for judges[cite: 25, 26, 27].

Challenges we ran into

  • Cloud Billing & API Quotas: Creating a Google Cloud billing account failed with an unresolvable OR_BACR2_59 error, locking us out of paid Gemini, Vertex AI, and Cloud Storage[cite: 26, 27]. We pivoted to a free-tier model chain with automatic fallbacks, deterministic SHA-256 caching, and local storage abstractions, ensuring the app runs flawlessly on a zero-dollar budget[cite: 26, 27].
  • Sandboxing Python Safely: To prevent calculation drift, we required the AI to write solution code for numerical questions. However, running AI-generated Python is dangerous, and we measured rogue snippets allocating memory at 1.8 GiB/s[cite: 27]. We solved this by implementing an AST whitelist, overriding mathematical operators with size guards, and enforcing a strict 512 MiB memory cap via Windows Job Objects / Linux RLIMIT_AS[cite: 26, 27].
  • Coarse Video Citations: Mapping transcripts to 60-second chunks was great for search but terrible for citations (videos started 11+ seconds late)[cite: 27]. We built a local embedding refinement step that compares the citing paragraph with overlapping 2-segment Whisper windows, successfully pinpointing YouTube links to the exact moment an example begins[cite: 25, 27].

Accomplishments that we're proud of

We built a rigorous evaluation testset (Task S9) and achieved exceptional benchmarks[cite: 26, 27]:

  • Context Key-Term Recall: 92.3% (Target: $\ge 80.0\%$)[cite: 25, 27]
  • Citation Recall: 81.5% overall, including 100.0% accuracy on lecture slides[cite: 25, 27]
  • Refusal Compliance: 100.0% isolation of out-of-domain queries[cite: 25, 27]
  • Latency & Cost: Cached response latencies of p50 = 0.2s while burning 0 live Gemini API tokens during evaluation[cite: 25, 27].

We are also proud of our custom PDF integration, which streams partial bytes via HTTP Range requests (206 Partial Content) and successfully resolves 2x2 slide grid metadata into displayed PDF coordinates to draw precise bounding boxes around cited slides[cite: 25, 27].

What we learned

We gained deep experience implementing Bayesian Knowledge Tracing (BKT) from scratch. We learned how to formally model lucky guesses (e.g., 0.25 for MCQs, 0.05 for numerical) and careless slips, and implemented a dynamic 14-day exponential forgetting decay formula ($p = 0.3 + (p - 0.3) \times 0.5^{\text{days}/14}$) that applies lazily at read-time without rewriting database history[cite: 25, 27]. We also learned the importance of server-authoritative state reconciliation to prevent UI race conditions when updating learner mastery[cite: 25, 27].

What's next for Groundwork · Multimodal AI Study Companion

While we completed our protected scope ahead of the feature freeze, our roadmap includes:

  1. Automated Slide Extraction: Adding LibreOffice headless conversion to support raw .pptx and .docx student uploads[cite: 25, 31].
  2. Visual Content Understanding: Describing keyframes and textbook figures using Gemini Vision models for full multimodal retrieval[cite: 25].
  3. Diagnostic Onboarding: Implementing a diagnostic quiz mode that estimates initial BKT priors for new students without requiring manual checkbox ticking[cite: 25, 27].

Course Material Attribution & Open Licensing

  • MIT OpenCourseWare 6.041 / 6.431 (Prof. John Tsitsiklis, Fall 2010) — Licensed under CC BY-NC-SA 4.0[cite: 25, 27].
  • Introduction to Probability (Charles M. Grinstead and J. Laurie Snell, AMS) — Licensed under GNU Free Documentation License (GFDL)[cite: 25, 27].
  • Attributions and open licenses are displayed natively on every source card in the UI[cite: 25, 27].

AI Assistant Transparency Disclosure

  • Architecture, Planning & Implementation: Gemini 3.8 Flash (via Google Antigravity)[cite: 25, 26].
  • Plan Reviews, Verification & Audits: Claude Code (Claude 3.7 Sonnet)[cite: 25, 26].
  • All engineering decisions, acceptance criteria, and final Git commits were governed and executed by Shreyas[cite: 25, 26].

Built With

Share this project:

Updates

Submission history