Sona

A voice-first AI math tutor that talks and draws on a live whiteboard in real time — diagrams, graphs, and equations, just like a real teacher. Speak. See. Understand.

Comment

Inspiration

Every student learns differently, but most AI tutoring tools today are text-based chat interfaces — you type a question, you get a wall of text back. That's not how real teaching works. A great math teacher talks to you, draws on the board, writes equations as they explain, and adapts in real time to your reactions. We wanted to build exactly that experience: an AI tutor you can simply talk to, that teaches on a live whiteboard just like a human would.

The Gemini Live API's bidirectional audio streaming made this possible for the first time — real-time, voice-first interaction where the AI can listen and speak naturally while simultaneously drawing on a shared canvas.

What it does

Sona is a voice-first AI math tutor with a collaborative whiteboard. Students speak naturally to Sona, and it responds by talking back while drawing diagrams, plotting graphs, writing equations, and annotating step-by-step solutions on a shared canvas — all in real time.

When a student asks about a new concept, Sona follows a structured whiteboard teaching flow:

  1. Draws the relevant diagrams or shapes
  2. Writes the key equations and rules in LaTeX
  3. Annotates and highlights important parts
  4. Speaks alongside each step, synchronized like a live lecture

Students can interrupt Sona at any time using a Hold to Talk button, giving them natural control over the conversation flow without ambient noise triggering interruptions.

How we built it

Sona is built as a microservices architecture on Google Cloud Run with four services:

  • Frontend Service — React + TypeScript + Konva.js canvas for the interactive whiteboard
  • Orchestrator Service — FastAPI + Google ADK + Gemini Live API handling bidirectional voice streaming and AI reasoning
  • Drawing Service — FastAPI + WebSocket streaming + a custom stroke DSL that executes tool calls as real-time canvas operations
  • Session Service — FastAPI + Firestore + Cloud Storage for session lifecycle and persistence

The AI brain uses Google ADK with the Gemini Live API in bidirectional streaming mode. We designed a semantic tool API that consolidates 14 internal drawing functions into 4 high-level tools — draw(), edit_canvas(), graph(), and highlight() — each with an action parameter. This dramatically reduced hallucinated and duplicate tool calls from the LLM.

A smart cursor system handles automatic element placement on the canvas, so the model never needs to calculate raw coordinates. It simply says "draw a triangle" and the cursor places it at the next logical position, with directional flow control (below, right, left, below_all).

Authentication is handled via Firebase Auth with JWT verification across all services, and the Drawing Service persists canvas state to Firestore.

Challenges we ran into

The biggest challenge was the infinite loop bug. Google ADK automatically feeds FunctionResponse events back into the model context after tool calls, which caused Gemini to continuously generate new tool calls in response to its own results — an infinite generation cycle. We solved this with custom turnComplete logic that suppresses tool results within the same turn and injects canvas state as fresh context on the next user input, breaking the auto-feedback cycle.

Audio distortion during real-time streaming was another significant challenge. Raw audio chunks from the Gemini Live API would arrive at uneven intervals, causing choppy playback. We implemented an audio buffering layer using AudioWorklet that accumulates chunks before playback, ensuring smooth and uninterrupted voice output.

Getting the LLM to draw accurately was also difficult. Early versions would approximate graphs with freehand strokes or place elements on top of each other. We addressed this by building deterministic backends — plot_function computes curve points server-side rather than trusting the LLM — and by designing the cursor system to remove coordinate math from the model's responsibilities entirely.

Accomplishments that we're proud of

  • 14 → 4 tool consolidation: Reducing the tool surface area made Gemini significantly more reliable. Fewer tools means fewer hallucination opportunities and cleaner action selection.
  • The cursor system: A deceptively simple idea that solved a hard problem. The model just says "next: right" and the system handles all the spatial layout math.
  • turnComplete logic: Finding and fixing the ADK infinite loop required deep understanding of the framework internals. Our solution is clean and generalizable to other ADK agents with tool use.
  • Real-time synchronized teaching: Watching Sona write a title, draw a triangle, label its sides, write the Pythagorean theorem in LaTeX, and explain it all through voice — simultaneously — is genuinely satisfying.

What we learned

  • Tool API design matters more than prompt engineering for agent reliability. Consolidating tools and making them semantic (action-based dispatch) reduced errors far more than prompt tweaks.
  • LLMs should not do math — deterministic backends for coordinate calculations, function plotting, and spatial layout are essential. Every time we trusted the model with numbers, it got creative in ways we didn't want.
  • Voice-first changes everything about how you design an AI interaction. Latency budgets are tighter, turn-taking logic is critical, and the user experience is far more sensitive to small glitches than in text chat.
  • ADK is powerful but opinionated — understanding its internal event loop (especially around FunctionResponse auto-injection) is crucial when building agents with complex tool interactions.

What's next for Sona

  • Multi-student sessions — allow a teacher and multiple students to share the same whiteboard with role-based permissions
  • Session replay — record and replay entire tutoring sessions (voice + canvas) for review
  • Adaptive difficulty — track student understanding across sessions and adjust explanation depth automatically
  • Broader subject support — extend beyond math to physics, chemistry, and any subject that benefits from visual explanation
  • Mobile-first experience — optimize the canvas and voice interaction for tablet use in classrooms

Built With

Share this project:

Updates

Submission history