Speaker Sense AI

PeakPulse turns your camera into an instant AI coach, delivering real-time posture analysis and instant audio feedback to keep your form locked in—preventing injury before it happens.

Comment

Inspiration

The primary spark behind this project came from observing tools like Flock—specifically how a live, continuous camera feed can be transformed into a rich source of real-time intelligence. We realized that live multimodal analysis could be applied to personal body language and posture during presentations, interviews, or daily work. Rather than relying on static image uploads, we set out to build PostureSense AI: a platform that streams real-time video and PCM audio to an AI model to evaluate body posture on the fly and deliver instant, actionable coaching cues alongside a post-session summary report.

How We Built It

We organized the project into three distinct layers to ensure full modularity and seamless integration across our four-person team:

  • Frontend (Next.js & React): Built a dynamic LiveHUD component that captures webcam frames (as base64 JPEGs) and streams raw 16kHz PCM audio via browser AudioContext.
  • Backend Gateway (FastAPI & WebSockets): Created an asynchronous proxy using FastAPI WebSockets that streams incoming video frames and audio chunks directly into the Gemini API without disk-latency overhead.
  • AI & Coaching Layer (Gemini Live API): Configured Gemini’s real-time multimodal capabilities with custom function calling (tool use) to output structured JSON events (LiveTick and SessionSummary).

Key System Formulation

To maintain strict real-time telemetry processing, we modeled the frame ingest rate and WebSocket payload queue as a function of target frame rate ($FPS$) and image data resolution ($S_{frame}$):

$$B_{required} = FPS \times (S_{frame} + S_{audio})$$

Where $S_{frame}$ is the base64-encoded JPEG size (~30–50 KB) and $S_{audio}$ represents the 16kHz PCM 16-bit audio chunk payload size emitted by the browser every 256ms.

Challenges We Faced

  • Frontend to Backend WebSocket Relay: Synchronizing bi-directional WebSocket frames between the Next.js client, FastAPI, and the Gemini API without introducing audio-video latency or buffer overflow.
  • JSON Parsing & Data Extraction: Handling structured tool calls and stream chunks from Gemini. Parsing dynamic function calls into typed schema models on the fly required building a resilient stream parser (JsonStreamParser).
  • Git Version Control & Merge Conflicts: Coordinating parallel feature branches across UI design, socket handling, and backend Gemini integration. We resolved complex merge conflicts by enforcing strict PR reviews and component boundaries.

What We Learned

  • Agentic Multimodal Workflows: How to leverage Gemini's Live API for concurrent audio and video ingestion to generate live structured cues and comprehensive session reports.
  • Teamwork & Task Delegation: How to divide tasks efficiently across frontend, backend, and AI pipeline layers using GitHub issues and branching strategies.
  • Browser Audio Engineering: Capturing raw audio buffer streams using the Web Audio API (ScriptProcessorNode / AudioWorklet) and converting float32 samples to 16-bit PCM binary payloads in real time.

Built With

  • ai-coach
  • asyncio
  • body-language
  • computer-vision
  • developer-tools
  • fastapi
  • gemini-api
  • google-genai
  • hackathon
  • health-tech
  • live-video-analysis
  • multimodal-ai
  • nextjs
  • pcm-audio
  • posture-coaching
  • pydantic
  • python
  • react
  • real-time-audio
  • speech-synthesis
  • tailwinds
  • typescript
  • webcam-stream
  • websockets
Share this project:

Updates