Main Tutoring Interface — Ready to Tutor
Gemini Analyzing Homework — Frame Detection
Step-by-step Explanation with Visual Aid
Live Agent Dashboard — Ready to Speak
Live Agent Mode — Real-time Voice Interaction

VisionTutor AI — Devpost Writeup

Inspiration

Every student has experienced that frustrating moment late at night — staring at a homework problem with no one to ask. Tutors are expensive, parents aren't always available, and typing questions into a chatbot feels nothing like actually being taught.

We were inspired by the idea of a tutor that doesn't just read your question — but actually sees your work, hears your confusion, and responds the way a real teacher would: naturally, instantly, and with patience.

When we discovered the Gemini Live API with its real-time audio and vision capabilities, we knew we had found the right tool. The ability to simultaneously see a student's camera and hear their voice — and respond within milliseconds — opened the door to something that feels genuinely different from existing AI tools.

What it does

VisionTutor AI is a real-time AI tutor that sees your homework through your webcam and explains it step by step in a natural voice.

It has two modes:

Standard Mode — The student points their camera at homework and clicks Analyze. Gemini Vision identifies the subject, extracts the problem, and generates a structured teaching plan. The AI explains each step, generates visual SVG diagrams to illustrate concepts, and creates similar practice questions for the student to try. Students can ask follow-up questions using their microphone at any point.

Live Mode — Powered by the Gemini Live API. The student turns on their mic and camera, and a real-time bidirectional session begins. Gemini sees the camera feed and hears the student simultaneously. There are no buttons to press — just natural conversation. The student can interrupt Gemini mid-sentence, ask follow-up questions, or show new problems. Gemini responds in a natural voice in under a second.

How we built it

Frontend: Next.js with TypeScript. We built custom React hooks for WebSocket management, webcam capture, Web Audio API integration, and the Gemini Live session. The UI has two distinct modes with real-time status indicators showing when Gemini is speaking, listening, or being interrupted.

Backend: FastAPI with Python. We built a multi-agent pipeline using four specialized agents — a Vision Agent that analyzes camera frames using Gemini Vision, a Reasoning Agent that creates structured teaching plans, a Teaching Agent that generates explanations and diagrams, and a Dialogue Agent that handles natural voice interruptions.

Live API Integration: The most technically complex part. We proxy the browser's microphone audio (16kHz PCM16) and camera frames (JPEG at 2-second intervals) through a FastAPI WebSocket to the Gemini Live API using the new google-genai SDK. Gemini's audio responses (24kHz PCM16) are streamed back and played through the Web Audio API in real time.

Diagram generation uses a two-step approach: Gemini returns diagram data as structured JSON, and we render it as SVG in Python — ensuring complete, reliable visuals every time.

The entire backend is deployed on Google Cloud Run, with Firebase Firestore handling session persistence. Infrastructure is provisioned with Terraform and deployment is automated via a single shell script (cloud_run_deploy.sh).

Challenges we ran into

WebSocket routing with FastAPI: FastAPI does not reliably apply router prefixes to WebSocket routes. We spent significant time debugging why the frontend was connecting but messages were never received — the fix was to declare the full path directly on the route decorator rather than relying on the prefix.

Gemini model availability: The google-generativeai SDK and the newer google-genai SDK list different models, and live-capable models do not appear in list_models() output. Finding the correct model name for bidiGenerateContent required testing multiple model strings.

SVG diagram truncation: When asking Gemini to generate raw SVG code, the model would truncate the output mid-element — producing blank or partial diagrams. We solved this by switching to a two-step approach: ask Gemini for diagram data as JSON (small and always complete), then build the SVG ourselves in Python.

Real-time audio pipeline: Getting the browser's microphone audio into the correct format (16kHz mono PCM16) and playing back Gemini's 24kHz audio responses without gaps or glitches required careful AudioContext management and a playback queue.

API quota exhaustion: During development we burned through free-tier quotas across multiple Google Cloud projects. This taught us to implement caching, reduce token counts in prompts, and add timeouts to every Gemini call.

Accomplishments that we're proud of

Getting the Gemini Live API working end-to-end — simultaneous audio and video streaming between a browser and Gemini with sub-second response latency — is something we're genuinely proud of. It required solving real-time audio encoding, WebSocket proxying, and playback synchronization all at once.

The multi-agent architecture works robustly even without Firestore credentials — every agent has an in-memory fallback so the app runs fully in local development without any cloud setup.

The interruption handling is seamless. A student can cut Gemini off mid-explanation and immediately ask a different question — exactly like interrupting a real tutor — and the system handles it gracefully without getting confused.

The visual diagram system generates clean, educational SVG diagrams reliably. By separating diagram data generation from rendering, we eliminated an entire class of bugs that plagued earlier versions.

We shipped a complete, working product — not a demo — with two fully functional tutoring modes, voice I/O, camera vision, diagram generation, practice questions, session management, and a live Cloud Run deployment.

What we learned

The Gemini Live API is genuinely different from standard LLM APIs. It is not a request-response model — it is a continuous stream where both sides can send and receive simultaneously. Designing around this requires a completely different mental model compared to typical AI integrations.

Real-time audio in the browser is harder than it looks. The Web Audio API, ScriptProcessorNode, AudioContext sample rates, and PCM encoding all interact in subtle ways. Getting clean 16kHz audio to Gemini required understanding the entire pipeline from microphone hardware to WebSocket payload.

Multi-agent systems need careful error boundaries. When one agent fails silently, the whole pipeline appears to hang. Adding granular logging and timeouts at every stage transformed debugging from guesswork into a systematic process.

Prompting for structured JSON output requires careful token budgeting. Long explanation fields in a JSON response can cause truncation that breaks the entire parse — keeping all string values short and raising token limits are both necessary.

Ship early and debug with real data. Many of our hardest bugs only appeared with actual camera frames and real microphone audio — they were impossible to reproduce with synthetic test inputs.

What's next for VisionTutor AI

Subject-specific tutoring modes: Specialized agents for math (step-by-step equation solving with LaTeX rendering), chemistry (molecular diagrams, balancing equations), and essay writing (grammar feedback, argument structure analysis).

Student progress tracking: Using Firestore to build a learning history — tracking which problems a student has struggled with, what concepts they've mastered, and generating personalized review sessions.

Classroom mode: A teacher dashboard where multiple students can connect simultaneously, with the AI routing questions intelligently and the teacher able to monitor all sessions.

Mobile app: The core Live API integration works on mobile browsers, but a native iOS/Android app would allow better camera control, offline caching of explanations, and push notifications for study reminders.

ADK integration: Migrating the backend agents to Google's Agent Development Kit for more robust orchestration, better tool use, and easier deployment to Google Cloud Run.

Whiteboard mode: Instead of just the webcam, students could share their screen or draw on a virtual whiteboard while talking to the AI — enabling support for digital homework assignments.