ORATOR.AI — Live Multimodal Public Speaking Coach

• 

• 

• 

• 

• 

Inspiration

Public speaking tools often fall into two extremes: static feedback after the fact, or generic AI tips that don’t adapt to what’s happening in the moment. While experimenting with Gemini Live APIs, I realized that low-latency multimodal reasoning opens a new category of tools — ones that can observe, react, and explain in real time.

The inspiration for ORATOR.AI came from a simple question:
What if an AI coach could watch and listen while you speak, not just analyze a recording later?

This project was motivated by the challenge of turning raw multimodal signals (audio, video, language) into actionable, real-time feedback without overwhelming the speaker.

What it does

ORATOR.AI is a live, multimodal public speaking coach.

During a speaking session, it:

• Streams microphone audio to Gemini Live for real-time transcription and audio reasoning
• Analyzes speech pace using word-level timestamps
• Tracks eye contact using on-device computer vision
• Visualizes live speaking metrics without interrupting the speaker

After the session, ORATOR.AI:

• Uses Gemini to generate a structured, natural-language critique
• Converts that critique into spoken feedback using text-to-speech
• Stores session summaries locally and displays trends in a dashboard

The goal is not just feedback, but reflection, helping speakers understand how they performed and where to improve.

How we built it

The system is built as a hybrid architecture:

• Gemini Live API is used for low-latency, bidirectional audio streaming and transcription
• Gemini reasoning models generate post-session critiques
• Text-to-Speech produces an audio review of the feedback
• MediaPipe runs locally in the browser to detect face presence and estimate eye contact
• Web Audio API calculates live volume levels
• Session metrics are visualized using interactive charts, and summaries are persisted in browser localStorage

This approach keeps latency low while ensuring Gemini is central to the reasoning and feedback loop.

Challenges we ran into

One of the biggest challenges was balancing real-time performance with meaningful feedback. Live systems leave very little room for delay, especially when audio and video are involved.

Another challenge was working within sandboxed environments where common browser features (like modal dialogs) are restricted, requiring custom UI solutions. We also had to be very intentional about distinguishing between metrics computed from real signals versus those simulated to demonstrate interaction design.

Finally, designing feedback that helps without distracting the speaker required multiple iterations.

Accomplishments that we're proud of

• Successfully integrating Gemini Live for real-time audio reasoning
• Building a working multimodal pipeline that combines audio, vision, and language
• Delivering post-session feedback as both text and synthesized speech
• Creating a dashboard that helps users reflect across multiple sessions
• Maintaining transparency about what is live, computed, or simulated

What we learned

This project reinforced that real-time AI systems are as much about restraint as capability. Knowing when not to interrupt is just as important as knowing when to intervene.

We also learned how powerful Gemini becomes when used beyond chat — as an engine for live reasoning, explanation, and feedback. Multimodal signals are most useful when they are contextualized, not just measured.

What's next for ORATOR.AI — Live Multimodal Public Speaking Coach

Future iterations would focus on:

• Replacing simulated metrics with fully model-driven analysis
• Adding true cross-session learning and adaptive training plans
• Deeper video reasoning using Gemini’s multimodal capabilities
• Expanding from session feedback to long-term skill development

ORATOR.AI is a foundation for exploring how AI can move from static analysis to interactive, real-time coaching.

Built With

• gemini3
• google-ai-studio
• react
• typescript
• websockets