Speakify

Speak your mind

Comment

Project Description

Speakify is an interactive, browser-based public speaking platform that transforms intimidating public speaking practice into a joyful, confidence-building conversation.

The Problem It Solves

Public speaking is cited as one of humanity's most universal fears—glossophobia affects over 75% of people. Practicing in front of a mirror lacks realistic feedback, while practicing in front of real audiences induces acute anxiety and performance pressure. Most speaking tools are dry and sterile, focusing solely on cold analytics rather than making the speaker feel comfortable, supported, and heard.

Key Features

  • 3D Anime AI Companions (from X G.I.R.L.S): Interactive 3D VRM companions (Hana 🌸, Kira ⚡, Sora ☁️, and Una 🌙) rendered with @pixiv/three-vrm and WebGL, featuring dynamic eye tracking, synchronized eyelid blinking, and real-time lip-sync mouth animation.
  • Two-Way Voice Speaking Sessions: Practice talking with your actual voice using browser-native Web Speech recognition and listen to spoken character replies powered by speech synthesis.
  • Continuous Interest Learning & Memory: Companions extract your hobbies, passions, favorite shows, and foods during conversation, remembering them across sessions to weave thoughtful callbacks and personal references into future dialogue.
  • Humor & Stress-Free Banter: One-click prompt triggers like "Tell me a joke! 🎭", playful puns, and friendly teasing help dissolve speaking anxiety and build genuine rapport.
  • Interactive 3D Virtual Audience: A full 3D auditorium where audience members stand on stage, nod in agreement, react with floor spotlight halos, and take turns waving hello (👋) to welcome you.
  • Real-Time Speech Delivery Analytics: Measures speaking pace in words per minute, detects filler words ("um", "like", "you know"), evaluates pauses, and offers optional on-device posture tracking via MediaPipe Pose.
  • Live Dashboard Companion Overlay: Your active companion stays pinned directly on your dashboard, commenting on your latest scores and offering instant warm-up advice.

Inspiration

We wanted to tackle the emotional barrier to public speaking. Traditional speech coaches and sterile analytics dashboards often feel like an exam, reinforcing anxiety instead of relieving it.

We asked ourselves: What if preparing for a big presentation felt less like a test and more like hanging out with your favorite anime companion?

Drawing inspiration from VTuber culture, conversational anime companions, and interactive 3D virtual worlds, we designed Speakify to provide a judgment-free, encouraging, and vibrant space where practicing your voice feels engaging and fun.

What it does

Speakify offers two complementary practice experiences:

  1. 1-on-1 Personalized Companion Chat: An intimate, low-pressure conversational space with your chosen anime AI companion. You can converse freely using voice or text. The companion listens, cracks jokes, learns what you care about, and builds your conversational confidence.
  2. Virtual Auditorium Practice Drills: Step onto a virtual 3D stage in front of an anime audience. Practice structured challenges such as:
    • 2-Minute Challenge: 1 minute to prep, 2 minutes of impromptu speaking on an AI-curated prompt.
    • Public Speaking Session: Deliver your presentation to an audience that listens attentively and asks interactive Q&A questions afterwards.
    • 1-on-1 Discuss & Debate: Practice casual back-and-forth dialogue or defend an argument against a skeptical peer.

How we built it

  • Framework & Runtime: Built with Next.js 16 (App Router & Turbopack) and TypeScript 5.7 for end-to-end type safety.
  • 3D Graphics & VRM Avatars: Powered by Three.js, @react-three/fiber, and @pixiv/three-vrm 3.5. We rigged and posed the 3D VRM models from the X G.I.R.L.S project, orchestrating skeletal bone rotations for idle breathing, nodding, friendly waving, and stable pointer tracking.
  • AI & Personality Engine: Powered by Groq Cloud API running ultra-fast LLMs (openai/gpt-oss-120b and openai/gpt-oss-20b) via secure server-side Next.js route handlers. Groq writes in-character dialogue, extracts user interests on the fly, and generates personalized speech coaching feedback.
  • Voice & Lip-Sync Pipeline: Browser-native Web Speech API for both speech-to-text recognition and text-to-speech synthesis. We bridged audio amplitude through a reactive voiceBus to drive real-time VRM mouth viseme shapes (aa, ih, oh).
  • Computer Vision: Integrated @mediapipe/tasks-vision for on-device body posture and gesture tracking without streaming user camera video to external servers.
  • Deployment: Configured for instant deployment on Netlify with @netlify/plugin-nextjs.

Challenges we ran into

  • VRM Orientation & Gimbal Singularity Jitter: Initially, avatars faced away from the camera due to standard VRM 0.0 coordinate conventions. When rotating them 180 degrees, the standard Three-VRM VRMLookAt system suffered from floating-point gimbal flips, causing rapid eye twitching. We resolved this by decoupling eye tracking from the gimbal singularity and relying on smooth head and neck bone lerping.
  • Facial Expression Clashes: VRM emotional presets like happy and relaxed automatically pull upper and lower eyelids shut into narrow squint arcs. We customized expression blending by explicitly zeroing out squint morphs and using subtle surprised blendshapes to keep wide-open anime eyes with clean, synchronized 140ms blinks.
  • Realistic Lip-Sync Cadence: Synthesizing speech without audio graph access often produces static open mouths. We coupled speech synthesis word boundary events (onboundary) with multi-frequency syllable wave envelopes to produce natural, rhythmic mouth movement that snaps shut cleanly when speech concludes.
  • Audience Posing & Immersion: Creating a natural audience feel without awkward chair clipping led us to design an open 3D stage with natural standing idle poses and a staggered waving sequence where audience members greet the speaker one by one.

Accomplishments that we're proud of

  • Full WebGL 3D VRM Pipeline in Next.js: Successfully loading and rendering multiple animated 3D anime VRM characters directly in the browser with silky smooth 60 FPS performance.
  • Zero-Latency Feel with Groq: Sub-second character response times that make conversations feel organic, lively, and natural.
  • Genuinely Comforting User Experience: Combining humor, memory callbacks, and warm encouragement to turn public speaking practice into something users actively look forward to.
  • Privacy-First Architecture: All voice transcription, synthesis, and optional body tracking happen locally on the user's machine; only text prompts reach the Groq API.

What we learned

  • How to harness @pixiv/three-vrm humanoid skeletal hierarchies and blendshape expression managers inside React Three Fiber.
  • Best practices for combining Web Speech API events with procedural animation buses to achieve responsive lip-syncing without heavy external audio analysis libraries.
  • The psychological impact of avatar design in educational technology: character warmth and personality dramatically lower the inhibition barrier for hesitant speakers.

What's next for Speakify

  • Multilingual Anime Companions: Expanding character voice synthesis and dialogue to support Japanese, Spanish, French, and Mandarin practice sessions.
  • Custom VRM Model Uploads: Allowing users to import their own VRoid / VRM avatars to practice with their favorite digital personas.
  • Real-Time Pitch & Intonation Visualization: Adding interactive pitch contours to help speakers master vocal variety, enthusiasm, and comedic timing.
  • Multiplayer Virtual Auditoriums: Allowing real peers to join the 3D room alongside anime companions for collaborative study groups and mock presentation rehearsals.

Built With

Share this project:

Updates

Submission history