Synapse: Multimodal Adaptive AI Study Companion

A multimodal Socratic study companion with exact textbook & lecture provenance, Bayesian Knowledge Tracing, and verified adaptive assessments—eliminating hallucinations and spoilers.

Comment

Inspiration

Today’s generative AI chatbots are ubiquitous in higher education, but in academic STEM curriculums, they suffer from four critical failure modes:

  1. Unverified Hallucinations & Fabricated Citations: LLMs routinely invent plausible-sounding equations, theorems, and sources.
  2. Premature Answer Spoiling: When a struggling student asks for help, chatbots immediately output the complete solution, bypassing the cognitive struggle essential to deep learning.
  3. Loss of Physical Origin Provenance: Text-only chunking loses physical coordinates—students cannot verify where a concept actually lives in their professor's slides, lecture videos, or assigned textbook.
  4. Static, Amnesic Assessments: Quizzes are typically generated in a vacuum without tracking probabilistic mastery or prerequisite knowledge over time.

We built Synapse to solve this. Inspired by university office hours and cognitive science, Synapse is an autonomous, source-grounded multimodal study companion that pairs physical coordinate provenance with probabilistic Bayesian Knowledge Tracing and Socratic pedagogy.

What It Does

Synapse transforms university course materials (textbooks, lecture slides, video recordings) into an interactive, zero-hallucination learning ecosystem:

  • Zero-Preprocessing Multimodal Ingestion & Physical Provenance: Parses course slide decks, timestamped lecture recordings, and textbook chapters. Every extracted fact preserves exact coordinates down to the textbook page number, slide number, and video timestamp via the custom synapse:// protocol.
  • Deep Citation Media Viewer: Students can inspect interactive source cards, preview textbook pages, and play back synchronized lecture video clips seeking directly to the relevant second.
  • Strict Hallucination Boundary Guard: An out-of-scope confidence gating filter quarantines external or out-of-syllabus questions, delivering transparent 100% accurate refusals rather than making up answers.
  • Source-Grounded Socratic Guidance Engine: Instead of spoiling answers, Synapse generates multi-tiered pedagogical hint ladders and diagnostic questions, encouraging independent derivation.
  • Continuous Bayesian Knowledge Tracing (BKT): Learner mastery is probabilistically updated in real-time as students practice: $$P(L_t \mid \text{Obs}) = \frac{P(L_t) \cdot (1 - P(S))}{P(L_t) \cdot (1 - P(S)) + (1 - P(L_t)) \cdot P(G)}$$ $$P(L_{t+1}) = P(L_t \mid \text{Obs}) + (1 - P(L_t \mid \text{Obs})) \cdot P(T)$$ Learners are categorized into Mastered (( \ge 0.85 )), Emerging (( 0.50 - 0.84 )), and Focus Area (( < 0.50 )) with automatic remediation routing.
  • Dual-Pass Verified Adaptive Assessments: Generates MCQs, short answers, and numerical problems (e.g., Effective Memory Access Time). An automated blind solver verifies answer keys before presentation, and a SHA-256 fingerprint registry eliminates duplicate questions.
  • Interactive Topological Knowledge DAG: An interactive SVG graph renders prerequisite chains across 6 topological layers (computed via Kahn’s algorithm) color-coded by the student's live BKT mastery.
  • Spaced Repetition & Bilingual Revision: SuperMemo-2 (SM-2) scheduling powers active recall flashcards, complete with textbook citations and code-switched Hindi-English (Hinglish) conceptual breakdowns.
  • "The Oxford Monograph" Visual Identity: A custom academic editorial design system rejecting generic AI templates (warm parchment, terracotta brick, forest green, Newsreader serif, WCAG 2.2 AA compliant).

How We Built It

We engineered Synapse under a disciplined Hybrid Agile-DevOps workflow across 6 test-driven sprints:

  • Backend: Modular monolith in Python 3.11 with FastAPI, SQLAlchemy 2.0 (async), and Pydantic v2 data contracts.
  • Storage & Vector Search: PostgreSQL 16 with pgvector for semantic embeddings, alongside SQLite/Redis support and relational persistence across 12 domain entities.
  • Learner Modeling & DAG Algorithms: Custom Python implementations of Bayesian Knowledge Tracing (BKT) with numerical (\epsilon)-clamping and Kahn's topological sorting for acyclic prerequisite DAGs.
  • Frontend: Next.js 16 (App Router, Turbopack), TypeScript, and Tailwind CSS. Custom SVG canvas for topological DAG rendering and an HTML5 media citation player.
  • Evaluation & Benchmarks: Automated RAGAS evaluation suite against a 26-item gold dataset, alongside a synthetic 75-session Cohort Simulator measuring mastery velocity across student personas.
  • Platform Engineering: Multi-stage Docker Compose containers, GitHub Actions CI/CD running linting, strict typing, and test suites.

Challenges We Ran Into

  1. Multimodal Coordinate Alignment: Reconciling temporal video timestamps ((\text{seconds})), spatial slide numbers, and static PDF pages into a unified, queryable citation schema required building custom coordinate extractors and the synapse:// URI resolver.
  2. Numerical Instability in Continuous BKT: Standard recursive BKT equations can suffer from floating-point underflow or degenerate bounds ((0.0) or (1.0)) over long sequential streaks. We solved this with mathematical (\epsilon)-clamping and verified correctness using Hypothesis property-based testing.
  3. Preventing LLM Answer-Spoiling: General foundation models instinctively want to "solve" homework questions. We engineered a multi-tier Socratic hint ladder state machine that enforces conceptual questions first and only dispenses progressive hints upon repeated student attempts.
  4. Assessment Verification & Duplicate Elimination: LLM-generated math/systems problems occasionally have flawed answer keys. We built a dual-pass verification pipeline where an independent solver model blinds and re-derives the answer; if any discrepancy occurs, the question is discarded. SHA-256 prompt-response hashing guaranteed a 0.0% question repetition rate.

Accomplishments That We're Proud Of

  • 100% Quality Gate Pass Rate: 60 / 60 automated unit, integration, and property tests passing with zero flakiness.
  • Rigorous RAGAS Benchmark Scores:
    • Faithfulness: 1.0000 (Target (\ge 0.90))
    • Answer Relevancy: 0.9070 (Target (\ge 0.88))
    • Context Precision: 0.9528 (Target (\ge 0.85))
    • Context Recall: 0.9900 (Target (\ge 0.90))
    • Refusal Accuracy: 100.0% on negative out-of-scope controls with zero fabricated citations.
  • Empirical Student Mastery Gains: The simulated novice cohort demonstrated a +0.5698 mastery gain across 5 sequential sessions.
  • Zero Placeholder Data: Seeded with a complete undergraduate curriculum: MIT CS-301: Operating Systems & Systems Architecture (based on OSTEP).
  • Accessible & Aesthetic: Full WCAG 2.2 AA contrast compliance with an editorial design system.

What We Learned

  • Cognitive Models + LLMs = True Personalization: Generative AI alone lacks memory of human learning curves. Marrying classical cognitive psychology (Bayesian Knowledge Tracing and SM-2 spaced repetition) with modern multimodal LLMs yields a tutor that is provably adaptive, not just conversational.
  • Provenance Must Be Physical: In higher education, a text snippet is not enough. Providing the exact slide schematic, textbook page, and lecture second gives students and educators the trust needed to adopt AI.

What's Next for Synapse

  • Real-Time Multimodal Voice Tutor: WebRTC-based low-latency ((< 500\text{ ms})) spoken Socratic dialogue.
  • LTI 1.3 Canvas & Blackboard Integration: Exporting continuous BKT telemetry directly into institutional gradebooks.
  • Expansion Across Curriculums: Broadening from Systems Architecture into Organic Chemistry, Discrete Mathematics, and Multimodal Medical Education.

Built With

Share this project:

Updates

Submission history