Lighthouse

Inspiration

One of our members, Min Jegal, brought up experiences she had while teaching children with ADHD and Autism and how it affected the students' understanding of facial expression, social cues, and tonality of conversation. This inspired us to focus on a tool that could help people with disabilities to interpret conversation.

What it does

Lighthouse aims to solve the problem of interpreting tonality, social cues, and emotions in real time during video calls and meetings. Lighthouse is designed to be a supportive companion during virtual meetings—whether you’re using our full-featured desktop app or a minimal, integrated Zoom extension. With real-time feedback, the potential for customizable settings, and plans for personalized guidance, our tool empowers people to interpret social cues with confidence.

How we built it

We built the app using Python, OpenAI API, Whisper, and DeepFace. Our Zoom extension had to be sidelined as we didn't have the time or API funding to fully integrate it with the backend, although a placeholder frontend can be found in the repo on the startup of Zoom on web.

Challenges we ran into

As we were given only 30 hours, we were on a time crunch. There were plenty of times we had to pivot, experiment with different APIs, and pray they would work.

Accomplishments that we're proud of

We're proud of the proof of concept we developed overall, and we believe Lighthouse can help make every virtual interaction a little clearer and more comfortable.

What we learned

We learned to be more comfortable with our programming languages etc.

What's next for Lighthouse

There's massive room for expansion with Lighthouse, for all Teams-like applications. Ideally, it could integrate as an overlay with all of them. There's also room for expansion with the AI itself, as live, detailed, tonality detection is a relatively untapped market. In our research, we only found one extension for Zoom which had a similar function but was far less broad.

Thanks for reading! 😊