StepWise Tutor

A live AI math tutor that sees your homework, hears your voice, and teaches step-by-step on a real blackboard — powered by Gemini Live.

Comment

Inspiration

Every student who ever said "I don't get it" deserved a patient, one-on-one tutor who could actually see their work and explain it in their words. But great tutoring is expensive and inaccessible. We asked: what if Gemini's Live API — voice, vision, and tool calls, all streaming simultaneously — could fill that gap for a Grade 3 student struggling with multiplication at 10 PM?

What it does

StepWise Tutor breaks the "chat box" paradigm entirely. The student:

  1. Shows their notebook via webcam or photo upload
  2. Speaks naturally — asking questions, giving wrong answers, or just saying "I don't get it"
  3. Watches a live blackboard update in real time as the AI tutor writes each step
  4. Gets corrected gently when the tutor sees a mistake via snapshot check

The AI tutor is always live — interruptible mid-sentence, adapts its language to the grade level (Grade 1 uses simple words; Olympiad level uses proof-oriented notation), and guides through one micro-step at a time using Socratic questioning.

How we built it

The core innovation is a bidirectional multimodal pipeline using Gemini 2.5 Flash Live API via Google ADK:

  • Voice IN: Browser captures mic via AudioWorkletProcessor, downsamples to 16 kHz Int16 PCM → base64 → WebSocket → Gemini bidi stream
  • Voice OUT: Gemini streams 24 kHz PCM → AudioPlayer with gapless scheduling → AudioContext playback
  • Vision: JPEG snapshots (webcam or file) → Gemini multimodal vision for problem reading and mistake detection
  • Blackboard sync: Two custom ADK tools — update_blackboard (writes steps to the live board) and request_student_action (sends a "Your Turn!" prompt card) — with an audio-gating pattern that syncs board updates ~300 ms into spoken audio so students see and hear each step in perfect sync

The backend is FastAPI + Google ADK InMemoryRunner + LiveRequestQueue on Google Cloud Run (--timeout=3600 for persistent WebSocket). The frontend is React + TypeScript with KaTeX math rendering. Mistakes and session summaries are logged to Firestore.

Challenges we ran into

The hardest problem was audio double-speech — Gemini would speak a sentence, then call the update_blackboard tool, then replay the same sentence in the post-tool audio. We solved this with an audio-gating pattern: all audio is buffered until the first tool call fires, the pre-tool buffer is discarded, and only clean post-tool audio plays.

When we added the second tool (request_student_action), a new variant of the bug appeared: the ADK function-response cycle created a micro-turn where the gate had reset, flushing the buffer again. We fixed this with a suppress_unfenced_flush flag that carries across turn boundaries to discard the post-action silence turn.

Accomplishments that we're proud of

  • A genuinely interruptible, live voice+vision tutoring loop — students can speak at any time to redirect the AI
  • Audio/board sync: seeing the teacher "write" a step at the moment they say it creates a real classroom feel
  • Grade-adaptive language: the same system prompt serves Grade 1 through Olympiad level, adjusting vocabulary and notation automatically
  • Full IaC deployment: a single deploy.sh script deploys the entire backend to Cloud Run with secrets injection

What we learned

Gemini Live's tool call timing is non-deterministic — the model may speak partial sentences before or after calling a tool in the same turn. Building robust audio gating required treating the audio stream as a stateful buffer with multi-condition drain logic, not just a simple on/off switch.

What's next for StepWise Tutor

  • Persistent student profiles: track mistake patterns across sessions (Firestore already logs mistake tags)
  • Handwriting recognition: use Gemini Vision to read pencil-written steps directly from the blackboard photo
  • Classroom mode: teacher dashboard showing which students are stuck and on which step
  • Support for geometry, graphs, and proofs with a visual diagram tool

Built With

  • audioworklet
  • fastapi
  • firestore
  • gemini-2.5-flash
  • google-adk
  • google-cloud-run
  • katex
  • pydantic
  • python
  • react
  • secret-manager
  • typescript
  • vite
  • web-audio-api
  • websocket
Share this project:

Updates

Submission history