math-coach-agent

app screenshot

Inspiration Every math student makes mistakes — but not every student gets a patient tutor who walks them through why they went wrong. In many households, parents struggle to help with math homework: they may not remember the concepts, speak the language of instruction, or have time to sit down problem-by-problem. Private tutoring is expensive, and most AI tools just give the answer.

We wanted to build something different: a voice-based AI tutor that coaches students through their mistakes in real-time — not by telling them the answer, but by asking guiding questions, just like a great human tutor would. Combined with spaced repetition, the system ensures students don't just fix one mistake — they retain the understanding over time.

What It Does Math Coach Agent is a full-stack AI tutoring platform with a complete learning loop:

For Students: Solve → Coach → Verify → Review Solve — Students select a problem from their workbook and submit their answer (typed or photographed) Coach — If wrong, Gemini Live API starts a real-time voice coaching session. The AI doesn't lecture — it asks "What did you do in the first step?", waits for the student to respond, and guides them to discover the error themselves Verify — After coaching, a variant problem (same concept, different numbers) tests whether the student truly understood Review — SM-2 spaced repetition schedules past mistakes for review at optimal intervals For Teachers/Parents: Scan → Curate → Monitor Scan Wizard — Upload workbook pages → Gemini Vision extracts problems, answers, solution steps, and even generates SVG diagrams Dispute Resolution — Review and override AI grading decisions Progress Tracking — Monitor per-student mastery and review schedules How We Built It Architecture Layer Technology Voice Coaching Gemini Live API (gemini-live-2.5-flash-native-audio) via WebSocket Agent Framework Google ADK with per-session agent creation and dynamic instruction injection Photo Grading & OCR Gemini Vision for work photo analysis and workbook scanning Variant Generation Gemini Pro for generating same-concept problems with different values Backend Python FastAPI on Google Cloud Run Database Firestore (users, workbooks, answer keys, attempts, mistake cards, study records) Storage Google Cloud Storage with signed URLs for work/problem photos Frontend Next.js + TypeScript + shadcn/ui + Tailwind CSS Auth Firebase Authentication Audio Pipeline Web Audio API — 16kHz PCM capture, 24kHz playback, RMS-based noise gating Key Technical Decisions Per-session agents instead of a singleton: each coaching session gets a fresh agent with the specific problem context (student answer, correct answer, common pitfalls) injected into its instructions Strict turn-taking enforcement: Gemini Live tends to monologue, so we heavily engineered the prompt to "ask ONE question, STOP, LISTEN" — creating a true Socratic dialogue Hybrid grading pipeline: Text answers use deterministic normalization + LLM fallback; photo answers use Gemini Vision for error classification across 8 specific error types (sign errors, arithmetic mistakes, fraction reduction, concept gaps, etc.) Multi-locale support: 9 languages with LaTeX-to-speech conversion (e.g., \frac{2}{3} → "3분의 2" in Korean, "3分の2" in Japanese) Challenges We Faced

Taming the Voice Model's Turn-Taking The biggest challenge was making Gemini Live API behave like a tutor rather than a lecturer. By default, the model tends to explain everything at once. We iterated extensively on the system instruction — adding emphatic markers like "DO NOT continue speaking after asking a question. STOP and LISTEN." — until we achieved natural back-and-forth dialogue where the student does most of the thinking.
ADK String-Only Variable Limitation Google ADK's instruction templating ({var_name}) only works with string values. Lists and integers cause KeyError. We worked around this by manually formatting instructions via format_coaching_instruction() before creating each agent — a non-obvious solution that took significant debugging.
Real-Time Audio Pipeline Building a reliable browser-to-server audio pipeline required careful engineering: 16kHz PCM capture → WebSocket streaming → Gemini Live processing → 24kHz PCM playback. We implemented RMS-based volume gating to filter ambient noise, barge-in support for natural interruption, and smooth audio buffering to eliminate playback gaps.
Accurate Math Grading at Scale Math answers come in many equivalent forms: 0.5, 1/2, ½, 50%. Our grading pipeline normalizes answers through regex patterns, then falls back to LLM comparison for edge cases. For photo submissions, Gemini Vision analyzes the actual work process — not just the final answer — to classify the specific type of error the student made.

What We Learned Voice changes everything in education: Typing is a barrier for young students. Real-time voice coaching feels fundamentally different from text-based AI — more personal, more engaging, and more effective for guiding mathematical reasoning Pedagogical design matters as much as technical design: The SM-2 spaced repetition + variant verification loop means students can't just memorize — they must demonstrate transferable understanding Gemini's multimodal capabilities unlock new UX patterns: Combining Vision (photo grading), Live Audio (coaching), and Pro (variant generation) in a single product creates an experience that wasn't possible before these models existed What's Next PDF workbook support for easier content onboarding Analytics dashboard with concept-level mastery visualization Group coaching sessions for classroom use