Social Sense AI

Inspiration

Social anxiety affects millions of people, yet most existing solutions only offer help after stressful situations have already happened. We wanted to create something proactive: a system that supports people in real time while users are actively navigating social interactions. SocialSense AI was inspired by the belief that confidence can be built through practice, guidance, and repetition. Our goal was to make social growth feel less overwhelming by creating a private, supportive AI companion.

What it does

SocialSense AI is a real-time social confidence coaching platform built for immersive environments. Using Meta Quest headsets, the system allows two users to interact in a shared virtual space while receiving live AI guidance and feedback. The platform helps users improve communication skills, reduce anxiety, and build confidence through realistic practice scenarios. Features include live voice communication, avatar-based interaction, calming haptic breathing prompts, and an AI coach that offers support during and after conversations.

How we built it

Mobile App (React) We built the SocialSense AI mobile experience using React + Vite and Supabase for real-time database, authentication, and audio storage. Conversations are recorded via the Web Speech API and MediaRecorder API, then processed through AssemblyAI for speaker diarization and sentiment analysis. We used Google Maps API for safe zone mapping and OpenAI GPT-4 to power the AI coaching engine.

$$ \text{Conversation} \rightarrow \text{Voice Recording} \rightarrow \text{AssemblyAI Analysis} \rightarrow \text{AI Coaching Feedback} $$

VR Experience (Unity) The immersive side was built using Unity, the Meta XR SDK, and Photon Fusion to create a shared VR environment with low-latency voice communication and avatar interaction. We integrated Convai to power natural AI conversations and personalized coaching. Custom UI systems handle live prompts, breathing exercises, and post-conversation feedback.

$$ \text{User Interaction} \rightarrow \text{VR Environment} \rightarrow \text{AI Analysis} \rightarrow \text{Real-Time Feedback} $$

Challenges we ran into

One of our biggest challenges was integrating multiple complex systems within a short hackathon timeline. Building stable multiplayer VR interactions while maintaining smooth performance on Meta Quest headsets was difficult. We also had to ensure the AI felt supportive and natural rather than scripted. Designing UI that felt immersive but non-intrusive in VR required several rounds of testing and iteration.

Accomplishments that we're proud of

We are proud of creating a working prototype that combines VR, AI, and live social interaction into one seamless experience. We successfully built real-time multiplayer functionality, immersive calming UI features, and an AI coach that provides meaningful support. Most importantly, we created a project that addresses a real mental health challenge in a practical and innovative way.

What we learned

Through this project, we learned how to rapidly prototype across multiple emerging technologies, including VR development, multiplayer networking, and conversational AI. We gained valuable experience in designing for emotional comfort and accessibility in immersive environments. Most importantly, we learned that meaningful innovation happens when technology is built with empathy at its core.

What's next for Social Sense AI

Our next steps are to expand SocialSense AI into a more personalized and scalable platform. We plan to improve the AI coach’s emotional intelligence, add more real-world social scenarios, and integrate advanced feedback such as eye contact, body language, and speech analysis. Long term, we want SocialSense AI to become an accessible daily companion that helps people build confidence and stronger social connections.