EmotiArt

Inspiration

Kevin is a dear friend to all of us. We go on trips to the beach, laugh, and play games. However, he sees the world in a different way the rest of us do. Similar to his imagination being incomprehensible to us, our emotions are confusing to him. Often times, his autism limits his ability to understand us and leaves him confused. We do not want him to continue in an endless cycle of being isolated by his emotion ability, instead we want a tool that allows him to see our world.

What it does

Our project creates a bridge between textual, visual and verbal analysis. It works by detecting emotions from different inputs and creating unique artistic representations understandable to autistic people. For textual analysis, the user pastes an input of text and then generates art which will then show the emotion. For all of our analyses, we have seven broad categories: Happy, Sad, Angry, Excited, Calm, Overwhelmed, and Anxious. Our visual and verbal analysis are combined with a dynamic system that creates nuanced representations of emotions. Adding novelty, we acknowledged how many people can often express combinations of different emotions. Hence, our website analyzes how multiple emotions are portrayed inside one recording. Through speech to text, and webcam input, our model analyzes facial features and text sentiment analysis to create a final emotional landscape.

How we built it

EmotiArt combines facial emotion detection (face-api.js via WebGL) with speech transcription (Web Speech API) and a custom keyword-based text sentiment scorer. Every ~3 seconds during a live recording session, these three signal streams are synthesized into a dominant emotion with a confidence score. That emotion is then fed into a custom Canvas 2D art engine that generates emotion-specific shapes (waves for calm, triangles for angry, stars for excited, etc.), layered depth fields, animated gradient backgrounds, glow effects, and floating particles. The tools we mainly used are Python, Next.js 16, React 19, TypeScript, and Tailwind CSS, deployed on Vercel as a free website.

Challenges we ran into

The hardest challenge was creating a nuanced art representation of the emotions. Raw face-api.js confidence scores are naturally low, and naively combining them with speech and text scores produced multiple motion switches mid-session. We solved this with a frame-buffer averaging system that only resolves a final emotion after a full recording session, smoothing out noise from individual frames. Next, another challenges was the actual art. Autistic people have specific preferences and we strive to meet these. So, we went through multiple iterations to create vibrant, unique, and animated art that represents multiple emotions successfully.

Accomplishments that we're proud of

Our greatest accomplishment was our novel emotion sentiment analysis which allowed for combinations of multiple emotions. We took multiple features from speech, visual, and textual analysis and integrated everything into a final artistic representation. The art showcased illustrations that allowed for easy understanding and nuanced displays.

What we learned

We learned how to integrate video analysis into our code, and training the model to detect user vocals. One of the most novel aspects of what we built is the site’s ability to synthesize these signals into nuanced, blended emotions rather than just a single dominant one. This pushed us to think carefully about how different emotional expressions can conflict or complement each other, and how to represent that complexity visually in the art.

What's next for EmotiArt

The next step is blending Computer Vision beyond just facial expressions. We want to detect user hand movements and body language, which include things like open arms, tense posture, or expressive gestures, then use those as additional inputs that influence the art in real time. This would make EmotiArt a full body emotional canvas, a great way for people who express themselves differently or who find non-verbal communication more natural than words. Potentially, combining with the AR capabilities of Meta Glasses.