Cadence

• 

  system architecture

Inspiration

Hundreds of millions of people speak languages with no dedicated learning platform. Cebuano (Bisaya) has ~27 million speakers across the Philippines and is absent from every major app. Traditional language apps like Duolingo and Babbel focus on a handful of high-traffic languages and rarely offer real pronunciation correction. I wanted to build something for the rest, especially children of immigrants who aren't able to naturally practice their language.

What it does

Cadence is a real-time AI voice tutor that lets learners have a live spoken conversation with an AI that adapts to their level. It corrects pronunciation in real time (with webcam-based mouth tracking via MediaPipe), teaches using evidence-based pedagogy (graduated scaffolding, fluency vs accuracy modes, no-cold-test rule), and tracks progress with SM-2 spaced repetition. Languages are pluggable via filesystem language packs.

How we built it

• Audio pipeline: WebSocket to the Gemini 2.5 Flash Native Audio Live API.
• Tool system: Five Gemini function calls handle vocabulary tracking, semantic content search (pgvector embeddings), and UI display (word cards, pronunciation coaching, conversation context).
• Persistence: Supabase PostgreSQL stores learner profiles, per-language vocabulary with spaced-repetition fields, session logs, and curriculum content with 768-dim pgvector embeddings for semantic retrieval.
• Pronunciation coaching: Gemini writes dynamic coaching text. When camera mode is activated, MediaPipe FaceLandmarker tracks lip shapes and sends JPEG frames + mouth metrics to Gemini every ~2 seconds for visual feedback.
• Frontend: Next.js, Clerk for auth

Challenges we ran into

• Pronunciation coaching is hard unless the Live API speaks in the target language. When the tutor models a word natively, learners can hear and mimic the sounds. Describing pronunciation in English is far less effective.
• Background noise sensitivity. The audio pipeline picks up ambient conversation easily, triggering unintended turn-taking or confusing transcription.
• Pulling in the right context at the right time. Semantic search helps, but timing curriculum content to match the natural flow of conversation — without overwhelming the learner or interrupting momentum — requires careful prompt engineering and tool scheduling.
• Non-English transcription through an English-configured model yields garbled or phonetically approximate text, making it hard to log what the learner actually said.
• Duplicate Gemini turns after tool calls. Gemini sometimes generates two consecutive speech turns after function calls. We built three complementary suppression layers (server-side audio suppression, 10s dedup window, frontend 800ms debounce) to handle this gracefully.

Accomplishments that we're proud of

I let my friends and family try it out and they were generally impressed with the quality of the model, it's ability to teach in a way that felt natural, and the fact that they actually learned something new (even for native speakers!).

What we learned

• Real-time bidirectional audio streaming with Gemini's Live API is powerful but requires careful state management — audio backpressure, turn boundaries, and cleanup of imperative resources (WebSocket, mic stream, audio player) are the most common failure modes.
• Even the live api needed a bit of prompt engineering to give lessons in a natural way.

What's next for Cadence

• Better pronunciation correction of foreign words
• Adding support for languages with non-latin scripts
• Better interaction UX between the user and tutor

Built With

• cloudrun
• gemini
• googleembeddingapi
• nextjs
• python
• supabase