ReadTheRoom

Inspiration

Many individuals face challenges with “reading the room,” making everyday interactions more daunting than they should be. Inspired by our teammate with mild autism and our foreign friends who deal with this exact problem everyday, we created a solution that could help others feel more comfortable in social settings. Our app uses AI to scan a room and identify emotions, activities, and group dynamics to give users a clear understanding of their surroundings. Our goal is to turn complex social situations into meaningful learning opportunities.

What it does

ReadTheRoom is a website designed for mobile users that takes video input and summarizes the social situation. It analyzes people’s actions and demeanor to separate them into two categories: approachable and non-approachable. This is combined with reasoning behind categories (position, current task, etc) to help the user understand, and hopefully be able to apply the info for themselves in the future. The summaries and videos are saved so the user can learn from the interactions later.

How we built it

Frontend: The UI/UX is built with Next.js/TypeScript/Tailwind. We first searched for potential designs (figma, online examples, previous winners, etc), and implemented the one we liked the most.
Video analysis: The video recorded by the user is sent to an AI (Gemini) that extracts details about the scene. This info is passed into another prompt to provide info into natural language to put into our UI.
Video storage: Videos recorded on a device are sent to an external cloud storage (Supabase storage) so that the recording can be saved and replayed later.
ElevenLabs: Reads the LLM summary out loud to the user.
NeuralSeek: The location contexts (location-specific accepted behavior, i.e. SBU Rule Poster) are stored in NeuralSeek’s knowledge base and referenced whenever detected in the same context/location to provide a more accurate answer.

Challenges

AI Accuracy: Making sure analysis of the room didn’t have any false negatives (people that are clearly open to converse but marked as “Non-Approachable”) and especially false positives (people that are clearly busy but are marked as “Approachable”). This required heavy prompt engineering / giving examples.
Choosing Data Metrics: Figuring out which metrics would be the most useful to a user and structuring it with thoughtfully designed UI/UX.
Implementing NeuralSeek: Understanding how to use NeuralSeek (API calls, documentation).

Accomplishments

Successfully analyzed videos with correct schema using Gemini API.
Submit the generated transcript into Eleven Labs to create an audio recording.
Finalized a project after pivoting multiple times.
Iterated on and discussed different possible design implementations, coming across a mutual consensus on the best UI.

What we learned

Tailwind
Prompting (context, plan, implementation)
UI/UX Design
Audio integration (ElevenLabs)
Knowledge bases (NeuralSeek)

What’s next

Ray-Ban Meta Glasses

Integrate ReadTheRoom into Ray-Ban Meta smart glasses for effortless use.
Allows for instant one-tap activation without relying on a phone or earbuds.
Less awkwardness when using the app (not waving a phone around).

Translation

Introduce a built-in translator to assist non-native English speakers in conversations.
Provide real-time transcription and translation for multilingual environments.
Automatically detect and switch to the correct translation mode.

Better Analysis