Project Clarity

Inspiration

While many of us may not realize it, we are constantly using social cues, such as a subtle smile or a brief frown to help us understand one another. However, for many people with conditions like nonverbal learning disorder (NVLD) or autism spectrum disorder (ASD), it can be challenging to communicate with others as they may struggle to pick up on social cues. This can lead to misunderstandings, social anxiety, and challenges in forming meaningful relationships.

What it does

Project Clarity acts as a real-time social cue interpreter. It uses a webcam to analyze the facial expressions of the person you are interacting with and provide either textual or auditory feedback to help you understand the emotional context of the conversation.

How we built it

On the backend we used Python + Flask to host a service that captures webcam input and analyzes it with FER (Facial Emotion Recognition), a deep-learning library built with OpenCV and TensorFlow that detects faces in images and classifies their emotional state. The front end is built with HTML, JS, and styled with Tailwind CSS, which displays the live video feed and the emotional feedback. The front end is hosted on GitHub Pages, while the backend is hosted on Hugging Face.

Challenges we ran into

We faced challenges deploying the Flask backend to run our facial expression detection model due to dependency conflicts. Since many of the required packages demanded different Python or library versions, resolving these issues was difficult. We fixed the issue by creating a spreadsheet to track dependency versions and consulting the Python Package Index to align versions and deploy our backend.

Accomplishments that we're proud of

We were especially proud of successfully deploying our Flask backend to Hugging Face. Since none of us had prior experience working with backends, navigating the platform, configuring environment variables, and managing dependency versions proved challenging.

What we learned

We learned lots about deploying backends and managing complex dependencies. None of us had previously worked with so many libraries simultaneously, and coordinating the correct versions of Python and each package proved challenging.

What's next for Project Clarity

Scalability & Real-World Deployment

While our project is just a web-developed tool, we believe it could be deployed in other technologies. Below we have listed a deployment strategy:

Phase 1

The current application allows users to build their skills and confidence by analyzing faces in a controlled, low-stakes environment. It can be used in therapy sessions or for at-home practice.

Phase 2

The next step is a browser extension that integrates with video conferencing platforms like Google Meet, Zoom, and Microsoft Teams. The extension would overlay a discreet analysis panel directly onto the video call, providing real-time cues during online classes, remote work meetings, and virtual social events.

Phase 3

The ultimate vision for Project Clarity is integration with augmented reality glasses. Imagine receiving subtle, heads-up display cues about a person's emotional state during an in-person conversation. It would be a life-changing tool for those with NVLD.

Future Features Roadmap:

Tone of Voice Analysis: Incorporate microphone input to analyze vocal tones (e.g., excited, calm, concerned) for a more complete picture of social cues
Contextual Learning Prompts: Provide helpful, educational tips based on detected cues, such as, "This person is smiling. This often means they are happy or agree with you."