ReadTheRoom

Inspiration

Post-presentation feedback is notoriously polite and unreliable, people say "looks great!" even when they were bored, skeptical, or lost. We wanted an automated empathy engine: a way for creators to genuinely "read the room" through second-by-second facial reactions without invading user privacy, paired with an AI co-pilot to directly interrogate the data.

What it does

Read The Room is an end-to-end video reaction-analytics platform: Shareable Testing Portal: Creators send a test link to participants to watch a video while their reaction is captured. Client-Side Emotion Detection: Runs entirely in the browser using on-device computer vision to detect nuanced expressions (joy, surprise, confusion, sadness, and frustration) mapped to exact timestamps. 100% private—no raw video feed ever leaves the user's device. Interactive Analytics Dashboard: Synchronizes video playback with an emotional intensity timeline, emoji markers, and scrubbable reaction peaks. AI Co-Pilot (Gemini API): An integrated chat assistant with custom database tools that answers questions (e.g., "Where did the viewer lose interest?" or "What was the funniest moment?") and cites exact, clickable video timestamps.

How we built it

Frontend: Built with React 19, TanStack Start, TanStack Router, Tailwind CSS, and Recharts for real-time synchronized timeline scrubbing. On-Device Vision: Google MediaPipe Tasks Vision (@mediapipe/tasks-vision) running via WebAssembly with GPU acceleration to classify 52 facial blendshapes into emotion states at 60 FPS. Backend & Storage: FastAPI (Python) and SQLite storing sessions, time-series reaction vectors, and calculated engagement metrics. AI Agent: Powered by the Google Gemini API (google-genai) with structured function calling (get_summary, get_top_reactions, get_insights) to query empirical backend data with zero hallucinations. Fully containerized with Docker Compose.

Challenges we ran into

We originally planned to build around the Presage Smart Spectra API, but it expected a native C++ environment while our app ran in JavaScript/TypeScript. Emotion jitter & bad lighting: Early on, a slight squint looked like anger and looking down at the keyboard registered as sadness. Scrubbing & timestamp drift: Syncing webcam frames to video playback is easy until someone pauses, rewinds, or scrubs ahead.

Accomplishments that we're proud of

Achieving 60 FPS real-time facial expression tracking entirely in the browser with zero server video transmission, protecting participant privacy. Building a grounded Gemini tool-calling agent that navigates time-series data and cites exact video timestamps. Delivering a unified, responsive dark-mode dashboard deployed seamlessly via Docker Compose.

What we learned

Translating Facial Action Coding System (FACS) blendshape coefficients into qualitative emotional dimensions. Designing multi-tool agentic loops with the Gemini API and robust deterministic fallbacks. Orchestrating full-stack real-time data pipelines between WebAssembly client streams and FastAPI backends.

What's next for ReadTheRoom

Multi-Viewer Aggregation: Combining reactions from hundreds of viewers into unified audience heatmaps. Audio Cadence Correlation: Correlating speaker pitch and pacing with audience confusion or excitement spikes. Video Editor Integration: Exporting emotional markers directly into Adobe Premiere Pro and DaVinci Resolve timelines.