Inspiration
The moments that bring people together, like a wedding toast, a class presentation or introducing yourself to a room of strangers, are often the ones people dread most. Our team knows this firsthand: one of us spent two years in debate and four years presenting in STEM programs, and was dinged for posture again and again. Another has six years of public speaking experience and organized a TEDx event, coaching student speakers. Good coaching is expensive and rare, and practicing in front of a mirror tells you almost nothing. We wanted a practice audience that actually reacts.
What it does
All Ears is a practice audience made of Earnest, an on-screen bunny, and a small Arduino robot.
While you speak, it watches and listens. If you slouch, sway, put your hands in your pockets, fidget or look away for more than two seconds, you get a toast, Earnest reacts, and the robot flashes red, beeps and names the problem on its screen.
When you finish, you get a report built from real measurements: words per minute, every filler word ("um," "uh," "like," "you know"), long pauses, speaking time, and a body-language tally including how often you faced the audience. Gemini turns those numbers into scored, specific feedback.
Then the robot raises its hand. Call on it and Earnest asks one sharp audience question: it picks your most checkable claim, runs a web search, and challenges you with a real source. You record an answer and get rated from 0 to 10, with what you did well and what to try next.
For repeatable practice, you can download any session and replay it with live coaching. The alerts and robot reactions fire in sync with the video every time.
How we built it
- Hub: a FastAPI backend with a WebSocket event bus. The app, the robot and the services never talk to each other directly. Everything goes through the hub, so any part can drop out without breaking the rest.
- Speech pipeline: Deepgram (nova-2) transcribes audio or video, keeps filler words, and boosts the speaker's name and topic terms. Our own Python measures pace, fillers and pauses from word-level timestamps. Gemini scores the speech against a rubric using those measured numbers, returns schema-checked JSON, and falls back to another model when one is overloaded. A full speech goes from upload to saved feedback in about 4-6 seconds.
- Audience question: Gemini picks a claim, SerpApi runs one Google search (with social media filtered out), and Gemini writes a sourced question. A second endpoint transcribes and rates the answer.
- Vision: MediaPipe pose and face tracking runs in the browser and calibrates to the speaker's own upright posture. Each behavior has its own cooldown, so alerts stay useful instead of overwhelming.
- Frontend: React and Vite, with Earnest reacting to the same events as the robot.
- Database: Supabase Postgres stores sessions, metrics and feedback, with an in-memory fallback.
- Hardware: an Arduino Mega with an RGB LED, a 16x2 LCD and a buzzer. A Python bridge turns hub events into serial commands, reconnects by itself and finds the board automatically.
Challenges we ran into
- Coordinating four people in 24 hours: we agreed on one shared event contract early, so each part could be built against fake events before the others were ready.
- Hardware: upload failures, a laptop adapter the board wouldn't work through, and a garbled LCD. We fixed them one by one, and designed the system so Earnest could stand in for the robot if needed.
- Filler words: many transcription models silently drop "um" and "uh," which defeats the purpose. We tested models until we found one that keeps them, and boosted names it kept mishearing.
- No webcams left in the hardware lab: we used phones as cameras and built replay mode so vision can run on recorded video.
- Keeping secrets safe: GitHub's push protection caught an API key before it went public, and we fixed our setup immediately.
Accomplishments that we're proud of
- A full loop in seconds: speech to measured metrics to scored feedback to a sourced audience question to a rated answer.
- Feedback grounded in measurements: every tip points to something the system actually measured.
- A physical robot that reacts live during a speech.
- A modular design where the robot, app and AI services can each fail without breaking the rest.
What we learned
- Agreeing on interfaces before writing code saves hours of integration.
- Scoping to a clear MVP early is what got us to a working demo.
- Hardware takes longer than software, so plan fallbacks from the start.
- Simple, explainable rules on top of a strong pretrained pose model beat training our own model under time pressure.
What's next for All Ears
- Live audio coaching: streaming transcription to flag pace and filler words during the speech, not just after.
- More modes: debate, interview and wedding-toast practice, each with its own rubric, plus fact-checking across several claims.
- A moving robot: servo-driven ears and a physical hand raise.
- Progress tracking: history charts across practice sessions.
note: Render was added after the hackathon, ngrok was used to host during the competition, but we decided to switch to Render for long term deployment to preserve continued access
Built With
- arduino
- c++
- css
- deepgram
- fastapi
- git
- google-gemini
- html
- javascript
- mediapipe
- postgresql
- psycopg
- pyserial
- python
- react
- sqlalchemy
- supabase
- typescript
- uvicorn
- vite
- websockets