All Ears

Inspiration

The moments that bring people together, like a wedding toast, a class presentation or introducing yourself to a room of strangers, are often the ones people dread most. Good speaking coaches are expensive and hard to find, and practicing alone in front of a mirror tells you almost nothing. We wanted a practice audience that actually reacts, so you can walk into the real moment already confident. One of our teammates has organized a TEDx event and coached student speakers, and his experience shaped what the coach looks for.

What it does

All Ears is a bunny-eared practice audience that listens and watches while you speak. During your speech it reacts live: when you slouch or hide your hands, the robot [lights up red / beeps / shows a message on its screen] and our on-screen bunny mascot reacts too. When you finish, All Ears gives you a scorecard built from real measurements: your pace in words per minute, every filler word (“um,” “uh,” “like”), long pauses, and body-language stats. An AI coach then turns those numbers into specific, friendly feedback and a one-line tip. Sessions are saved, so you can track how you improve over time.

How we built it Backend: a FastAPI server acts as the hub. A WebSocket event bus broadcasts every event (state changes, alerts, feedback) to the app and the robot at once, so each part can be built and tested independently. Speech pipeline: audio goes to Deepgram (nova-2) with filler-word detection and word-level timestamps. We compute our own metrics in Python (pace, pauses, longest pause, speaking time, filler counts), then send them to Google Gemini, which returns structured JSON feedback scored against a rubric based on real speech-coaching practice. A full speech goes from upload to saved feedback in about 4–6 seconds. Database: Supabase Postgres stores sessions, metrics and feedback, with an automatic in-memory fallback if the database is unreachable. Vision: MediaPipe pose tracking runs in the browser, comparing your posture against a personal baseline to detect slouching, hidden hands and fidgeting. Frontend: a React app with an animated bunny mascot that reacts to the same events as the robot. Hardware: an Arduino Mega robot (LEDs, [LCD, buzzer, servo-driven ear]) connected over USB. A Python bridge translates hub events into simple serial commands. Challenges we ran into Coordinating four people in 24 hours. We agreed on one shared event contract early, so the frontend, hardware and backend could each be built against fake events before the others were ready. Hardware integration. Getting the Arduino to react reliably to live events took longer than planned, so we designed the system so the on-screen mascot could stand in for the robot. Venue limitations. The hardware lab’s webcams were gone, so we planned to use phones as webcams and built a replay mode that runs vision on recorded videos. Speech recognition details. Most transcription services silently drop filler words, which defeats the purpose. We had to find and configure one that keeps them, and handle names and rare words it misheard. Keeping secrets safe. GitHub’s push protection caught an API key before it went public, and we fixed our setup right away. Accomplishments that we’re proud of A full end-to-end pipeline, from a spoken speech to scored, saved AI feedback, in a few seconds. Feedback based on real measurements rather than vague AI opinions: every tip points to something the system actually measured. A modular architecture where the robot, app and AI can each fail or be swapped without breaking the rest. [The robot physically reacting to a live speech.] What we learned Agreeing on interfaces before writing code saves hours of integration pain. Scoping matters: cutting to a clear MVP early is what got us to a working demo. Hardware takes longer than software, so plan fallbacks from the start. Simple, explainable rules on top of a strong pretrained pose model beat training our own model under time pressure. What’s next for All Ears Live audio coaching: streaming transcription so the robot can flag pace and filler words during the speech, not just after. The bunny raises an ear: after your speech, All Ears challenges your weakest claim with a real, sourced counter-question, like a tough audience member. More modes: interview practice, debate and wedding toasts, each with its own rubric. Progress tracking: history charts that show improvement across practice sessions. Eye-contact detection and more expressive robot ears. Tracks : Meta, Serapi, Gemini, Hardware