Socially

Inspiration

One in five people struggle to read social cues. For people with autism, social anxiety, or hearing impairments, understanding what someone really means can be difficult and stressful. We wanted to build something that acts like a real-time interpreter and helps those who need it.

What it does

Socially is a real-time social cue translator. It uses a live camera feed to analyze facial expressions, body language, and tone over time, then combines that information to infer how someone is actually feeling. The system then generates live coaching, explaining what the other person is likely feeling and what you could say next. It's like a mentor based on real emotions and people.

How we built it

For analysis, we used models that were custom-trained based on data sets that could process facial expressions and temporal changes in mood. 2 models work in parallel on each frame for the best outcomes.

Challenges we ran into

One of the biggest challenges we ran into was trying to set up the AI Socially Coach. This coach needs to sync up with the transcript of the voice detector and the emotions of the person. Combining these two while trying not to make it laggy took a lot of effort and changes.

Accomplishments that we're proud of

We are proud of training an accurate model and creating a real social impact on people who can actually use this tool. This is one of the most complex programs that we have made as a group.

What we learned

We learned how to design and optimize real-time AI systems, especially when combining multiple models with different input types and time dependencies. We also learned that raw model outputs aren’t enough: interpretability and user experience matter just as much as accuracy.

What's next for Socially

We want to integrate this program into actual hardware like glasses, so that way, it's more accessible for users and they will have a little AI chatbot that can talk to them at all times.

Built With

• ai
• amd
• apis:
• bilstm
• claude
• coaching
• generation)
• hardware:
• languages:-python-javascript-/-typescript-frameworks-&-tools:-fastapi-websockets-mediapipe-machine-learning:-pytorch-(or-tensorflow-?-use-what-you-actually-used)-custom-cnn
• models
• transformer
• whisper