Zubro - voice-first English tutor

Talk, get feedback, improve. Zubro uses Gemini Live API to assess your level, generate visual aids, come up with fun exercises and adapt to your progress - all through real-time voice conversation.

Comment

Inspiration

I'm a Polish engineer who's been speaking English professionally for years. My English is fine - I give talks at international conferences, I write technical documentation, I hold meetings in English every day. But "fine" isn't "good." I still hesitate before certain verb tenses. I still catch myself using the same safe vocabulary instead of the word I actually mean. I still avoid sentences I'm not sure I can land.

Here's the thing: there are plenty of apps and courses for starting a language. Duolingo, Babbel, structured textbooks - they're great for beginners. That's not the gap I'm trying to fill. The gap is what happens after. You reach a level where you're comfortable enough to stop trying, but not fluent enough to stop needing practice. Language skills are like muscles - if you don't use them, they quietly get weaker. And at this stage, going back to structured lessons feels boring, repetitive, and frankly beneath where you are.

What I wanted was something different: a conversation partner. Not a teacher who forces every interaction into a lesson plan. A companion I could talk to about my day, brainstorm project ideas with, rehearse a stressful presentation in front of, or prepare for a difficult conversation with - and who would help me with my English when I asked for it, not on a rigid schedule. Someone patient, always available, and genuinely useful beyond just language learning.

When Google announced the Gemini Live API - real-time, bidirectional voice with native audio - I realized I could actually build this.

What it does

Zubro is a voice-first English learning companion. You talk to it. It talks back. No typing, no multiple choice, no textbook exercises.

Level assessment - an agentic workflow of four exercises (warmup conversation, free speech monologue, roleplay, describe a generated slide) that evaluates grammar, vocabulary, pronunciation, fluency, and comprehension. The agent picks up context between exercises - it generates the slide based on the hobby the user mentioned during the monologue, creating a seamless, personalized experience. Produces a detailed report card with animated score gauges and feedback.

Free conversation - open-ended voice practice where Zubro operates as a live agent with tools and user's learning artefacts at its disposal. It can generate images on demand, give real-time feedback on your English, and manage a personal practice list across sessions. Returning users get a personalized greeting that references their last session and pending practice suggestions. The conversation is fully interruptible (barge-in supported) just like talking to a real human.

Presentation coach - upload a PDF of your slides, then either practice presenting solo (with post-session evaluation) or have a live two-way discussion with Zubro about each slide. Zubro sees the current slide and responds accordingly.

Roleplay scenarios - practice realistic situations (job interviews, ordering coffee, rebooking a flight) with Zubro playing the other role. Pick a preset or write your own scenario, choose your time limit, and get detailed feedback after.

Guided onboarding - Zubro walks new users through the interface via voice, highlights UI elements, and runs a mic test. All scripted from the backend.

Adaptive learning - Zubro tracks progress across sessions, maintains AI-generated practice suggestions.

How I built it

The key architectural decision: the browser never talks to Gemini directly. A Python backend (FastAPI on Cloud Run) sits between the browser and the Gemini Live API. The browser is a thin audio pipe and UI layer. The backend is the agent - it controls everything - system instructions, tool definitions, evaluation pipelines, session lifecycle, and mic control.

This backend-in-the-middle design enabled the most interesting technical features:

  • Dual evaluation pipeline - two separate Gemini models run alongside the conversation without the user knowing. An OngoingEvaluator fires after each utterance for real-time scoring. A SessionEvaluator collects all audio and produces a comprehensive report once the session ends. The Live API model is free to just be a good conversation partner.
  • Session recovery - WebSocket connections to the Live API drop frequently (1008/1011 errors, GoAway rotations). LiveSession implements automatic reconnection with resumption handles, utterance replay, and a time-based retry budget. For the 10-minute assessment, progress is saved to Firestore after each exercise - if everything fails, the user can reload and resume.
  • Graceful shutdown - when a time limit hits, a three-way decision tree based on who is currently speaking ensures the conversation ends naturally, not abruptly.
  • Tool-generated media injection - when Zubro generates an image via the generate_image tool, it appears in the browser AND gets injected back into the Live API session, so Zubro can see and discuss what it just created.

Deployment is automated - setup.sh provisions the GCP infrastructure, cloudrun-deploy.sh deploys to Cloud Run.

Five different Gemini models power different parts of the system - native audio for voice, Flash Lite for evaluation, Flash for image generation.

Full technical deep dive: zubro-v1 readme.md and technical blog post Building Zubro - a voice-first English tutor on Google's Gemini Live API

Challenges I ran into

  • Live API WebSocket stability - Live API session, especially around tool calls trigger random crashes. I built a multi-layer recovery system, but it took significant iteration to get right - especially the utterance replay after reconnection.
  • Audio pipeline - getting smooth, gap-free audio playback across browsers and devices was harder than expected. Ring buffers caused fast-forward audio mid-sentence, setting AudioContext to 24kHz broke Firefox's echo cancellation entirely, and mobile playback was consistently worse than desktop. Took several iterations to get right.
  • Evaluation quality - a single Live API session can't effectively hold a natural conversation AND track grammar, vocabulary, and pronunciation at the same time. Asking one model to do both meant it either became a nitpicky teacher (bad conversation) or a friendly companion (unreliable scores). The breakthrough was separating these into independent pipelines - the Live API model just talks, dedicated Flash Lite models evaluate in parallel.
  • VAD sensitivity - the Live API's voice activity detection picks up background noise, keyboard typing, and even Zubro's own audio through speakers. I implemented three-layer mic control (server-side drop, backend-controlled mute, user-controlled mute) to manage this.

Accomplishments I'm proud of

  • It's a real product, not a demo. Assessment with progress recovery, daily usage quotas, account management, help page, mobile support, deployment automation - the kind of things that don't show in a 4-minute video but matter for actual users.
  • Production-grade architecture from day one - Firebase Auth with session cookies, Firestore for user data, GCS with auto-delete for temporary media, API keys in Secret Manager (never baked in), dedicated service account with least-privilege IAM roles, External Application Load Balancer with SSL certificates. No shortcuts that would need to be undone later.
  • The agentic conversation loop: "What's on my practice list?" → Zubro reads it → "Can you generate a visual aid for mixed conditionals?" → image appears → we discuss it. Three tools in one natural flow.
  • Non-disruptive message injection - injecting text into an active Live API session interrupts the model mid-speech, but the backend needs to push context (eval results, tool outputs, slide descriptions) without cutting Zubro off. Solved by coordinating with the model's turn completion before injecting - enables background tasks to feed results back into the conversation seamlessly.
  • Presentation coach - upload a PDF, then either practice presenting solo with per-slide audio recording and detailed evaluation, or have a live discussion where Zubro sees the current slide and talks about it with you. Two distinct modes, one feature.

What I learned

  • Session resumption is not optional - build it from day one. Connections will drop.
  • The model is surprisingly good as a persona - two sentences of system instruction produce a consistent, patient tutor. I expected to need heavy prompting.
  • Audio in, audio out - keep it simple - spend your complexity budget on what happens around the conversation (evaluation, state, recovery), not on the conversation itself.
  • Media injection just works - sending images into the live session is seamless. Non-disruptive text injection and graceful session endings? Those I had to figure out myself.

What's next

  • Writing coach - write or just paste your text and get voice guidance on what to fix and why, without Zubro rewriting it for you. You do the editing, you do the learning, Zubro see current state and guides you along the way.
  • Progress dashboard with improvement charts over time
  • Accent practice exercises
  • Explore expanding beyond English to other languages

Built With

Share this project:

Updates

Submission history