Wordzzz Buddy

main page
music list by level and language
karaoke mode: correct words
karaoke: flashcards
karaoke mode: wrong words
stories
story settings
story conversation
flashcards

Inspiration

Learning a new language is one of the most rewarding yet challenging journeys. While reading and writing skills can be developed through apps and textbooks, speaking fluency remains the biggest hurdle for millions of learners worldwide. The fear of making mistakes, lack of conversation partners, and limited opportunities for practice create a gap between "knowing" a language and actually "speaking" it.

We were inspired by a simple question: What if anyone could have a patient, intelligent conversation partner available 24/7, who adapts to their level, never judges mistakes, and responds naturally in milliseconds?

The breakthrough came when we realized that modern AI has finally reached the point where:

LLM inference can be fast enough for natural conversation (<500ms)
Text-to-speech sounds genuinely human
Speech recognition works reliably across 10+ languages

We combined these technologies with engaging learning modes — scenario-based stories, open conversations, and even karaoke-style singing — to create an immersive voice-first language learning experience.

What it does

Wordzzz Buddy is an AI-powered voice conversation platform with three engaging modes:

🗣️ Free Chat

Open-ended voice conversations on any topic. The AI:

Adapts vocabulary complexity to your level (beginner → advanced)
Corrects mistakes naturally by rephrasing in its response
Keeps conversations flowing with follow-up questions
Supports 10+ languages including English, Spanish, French, German, Japanese, Korean, Chinese, and more

📖 Interactive Stories

Immersive role-play scenarios where you practice real-world situations:

🍽️ Restaurant in Paris — Order food and interact with a French waiter
💼 Job Interview — Practice professional English for tech interviews
✈️ Airport Check-in — Handle travel situations with confidence
🏥 Doctor Appointment — Describe symptoms and understand advice
🔍 Detective Mystery — Solve crimes through investigation and questioning

Each scenario includes specific goals to complete and vocabulary to master.

🎤 Learn by Music (Karaoke Mode)

Sing along to real songs and get real-time pronunciation feedback:

Synchronized lyrics with word-by-word highlighting
Voice recording with confidence analysis
Color-coded results showing pronunciation accuracy

📚 Flashcard Training

Vocabulary review with interactive flashcards that track what you remember and what needs more practice.

How we built it

Architecture

Frontend: Next.js 15 + React 19 with Tailwind CSS (Glassmorphism UI)

Backend Services:

Deepgram Nova-2 — Speech-to-Text (Voice → Text)
Cerebras llama3.3-70b — LLM Brain (Ultra-fast inference)
ElevenLabs — Text-to-Speech (Text → Natural Voice)
LiveKit — Real-time voice infrastructure

Tech Stack

LLM: Cerebras (llama3.3-70b) — Ultra-fast inference (~100ms)

Speech-to-Text: Deepgram Nova-2 — Accurate transcription with word timestamps

Text-to-Speech: ElevenLabs — Natural multilingual voices

Real-time: LiveKit — Low-latency voice infrastructure

Framework: Next.js 15 App Router — Modern React with server components

Styling: Tailwind CSS — Beautiful glassmorphism design

Key Technical Implementations

1. Conversation State Machine

Manages turn-taking, silence detection, and natural interruptions with phases: idle → listening → processing → speaking → waiting.

2. Word-Synchronized TTS Playback

Each word highlights as it's spoken using timing data from ElevenLabs API.

3. Adaptive System Prompts

Dynamic prompts adjust based on difficulty:

Beginner: Simple words, slow pace, helpful translations
Intermediate: Natural idioms, gentle corrections
Advanced: Complex discussions, cultural nuances

4. Dual STT Strategy

Primary: Deepgram API for high accuracy
Fallback: Browser Web Speech API for resilience
Both run in parallel for best user experience

Challenges we ran into

1. The Latency Problem

Voice conversations feel awkward with delays over 1 second. We needed the entire pipeline (STT → LLM → TTS) to complete in under 500ms.

Solution: Chose Cerebras for ~100ms LLM inference, implemented streaming responses, and parallelized independent API calls.

2. Making AI Sound Natural

Early versions felt robotic — the AI gave long, lecture-like responses that killed conversation flow.

Solution: Extensive prompt engineering with rules like:

"Keep responses SHORT — 1-3 sentences max"
"Use natural fillers: 'well...', 'hmm...'"
"React before responding"
"NEVER break character"

3. Cross-Language Speech Recognition

Supporting 10+ languages with varying accuracy was complex. Some languages worked great, others struggled.

Solution: Hybrid approach using Deepgram as primary with Web Speech API fallback. We also record raw audio during Web Speech recognition to send to Deepgram for final transcription.

4. Karaoke Word Matching

Matching sung words to lyrics in real-time while handling different pronunciations, accents, and timing variations.

Solution: Normalized word comparison with fuzzy matching algorithm.

5. Browser Audio Permissions

Different browsers handle microphone access differently, especially on mobile.

Solution: Explicit permission flows, audio context management, and MediaRecorder with multiple codec fallbacks (WebM, MP4, WAV).

Accomplishments that we're proud of

✨ Sub-500ms Response Time — Conversations feel genuinely natural with Cerebras-powered inference
🎯 Word-by-Word Sync — TTS playback highlights each word as it's spoken, creating an immersive experience
🌍 10+ Languages — Full support for English, Spanish, French, German, Italian, Portuguese, Russian, Chinese, Japanese, and Korean
🎭 Immersive Scenarios — Role-play stories that make practice feel like a game, not a lesson
🎤 Karaoke Mode — A unique approach to pronunciation practice that's actually fun
💎 Beautiful UI — Glassmorphism design that looks stunning and feels modern
🔄 Graceful Degradation — Falls back to browser APIs when premium services fail, ensuring the app always works
📊 Real-time Confidence Visualization — Users see exactly which words they pronounced well and which need work

What we learned

Latency is UX — In voice applications, every 100ms matters. We learned to measure and optimize every step of the pipeline obsessively.
Prompt Engineering is Crucial — Small wording changes create dramatic differences in conversation quality. "Keep it SHORT" and "react naturally" transformed robotic responses into engaging dialogue.
Fallbacks Save Users — Building graceful degradation (browser APIs when premium services fail) ensures the app works everywhere, every time.
Multi-modal Learning Engages — Combining voice conversations, story scenarios, and music creates more engagement than any single approach.
State Management Complexity — Voice apps have intricate states (listening, speaking, processing, interrupted). A clear state machine prevents countless bugs.
Testing Voice is Hard — Automated testing for voice interactions is challenging. We relied heavily on real-user feedback loops.

What's next for Wordzzz Buddy

🎯 Phoneme-Level Pronunciation Feedback — Detailed analysis of specific sounds, not just words
📈 Progress Dashboard — Track vocabulary growth, fluency metrics, and practice streaks over time
👥 Multi-Character Conversations — Practice with multiple AI personas in group scenarios
🎮 Gamification — Points, achievements, daily challenges, and leaderboards
📱 Native Mobile Apps — iOS and Android apps with offline mode support
🤝 Community Stories — Let users create and share their own practice scenarios
🎵 Expanded Music Library — More songs across languages with difficulty ratings
🧠 Spaced Repetition — Intelligent vocabulary review based on forgetting curves