Mirror Mind

Inspiration When I was in 9th grade, there was a special needs kid named Kanishka that struggled with emotional expressions. When the teacher was talking to him, he didn’t know how to respond, and couldn’t control his facial expressions. With this real world problem, me and my team were inspired to create Mirror Mind

What it does You can pick a preset emotion to practice, like "happy" or "surprised." Or, you can also upload a video of a conversation and then have a custom emotion based off of the conversation. The app turns on your webcam and tracks your face live using facial landmark detection. As you try to make that expression, it generates a heatmap overlay on your face showing which areas are matching the target expression (green = good match, red = needs work). You get a live match score that updates in real time, a timer, and quick tips telling you exactly what to adjust — like "lift the corners of your mouth" or "relax your eyes

How we built it

So we started off with some basic front end stuff with the button and the camera feature and the life feed showing. Then once we got that working we got a Gemini API key and we implemented that to show as part of the face feedback. This process included things like modeling and training the AI, the different face muscles needed, and how they should be when there are different emotions. That led to a model being removed of the muscles that gets placed on your face on the Feedback life video and that then gets color coded as if the muscles needs to relax or stretch or tighten up depending on the emotion, The colors were an indication of this as green is good Yellow is stretch and Red is tighten, This led to many failures but we got it working through rigorous rate training and constant debugging of the model. We then used Desmos and other grading calculators to build an equation of the graph that shows the model's point and build a percentage of closeness to perfect facial emotion. This was then converted to a live feedback graph where the X is the time and the Y is the percentage. After all that was done we worked back on the Front end where we used AI to enhance our look and get new ideas for things we could better implement.

Challenges we ran into

One of the major challenges we ran into was mapping out all the muscles and having the AI know when to tell the live feedback which muscles needed to relax and tighten. To start to solve this problem we first started with 20 different pictures of people's faces to show the AI for all the different muscles in a face and this led to a very simple grip map of the face that closely adapts to each person's face. Then after that we got different points of five and added color coding to it so that each part had the ability to change colors. Then using different pictures of ourselves and some AI generated ones we were able to feed the AI with 6 basic emotions that it was accurately able to predict and correct. This led to yet another big challenge which was the video feature. In this the AI had to find the emotion the person was trying to make. This was hard because the AI was not able to find the emotion the person was trying to make. So we had to use videos of our own and use a preset to feed the AI through that we were able to train the AI a bit and help me adapt to the emotional use.

Accomplishments that we're proud of

We are really proud of the video training and building a model that then gets used to learn that emotion because we feel that it really targets the person's personal needs. This was also the hardest part of the Project and the part we spent the most time on as getting this to work needed lots of training and models that required down time and trial and error. This uncertainty left us with many questions and hopes that after many trials the video would finally work.

What we learned

We learned that using AI as an API key can help in many ways more than what we had previously imagined. Before, we thought it was something that was only going to being in the backend and could not be used for post processing display but through experimentation we learned that we could not only train the Ai model for our personal use but also use it for front end display for life feedback.

What's next for Mirror Mind

The next goal for Mirror Mind is to advance on these ideas we have already used, and if possible make an even more robust AI, or possibly enhance new tools and methods to make an even bigger impact

Built With

• css
• next.js
• pnpm
• postcss
• react
• scss
• typescript
• vite