Inspiration
We talk a lot about accessibility ramps, screen readers, and closed captions — and those things matter enormously. But there is a whole category of disability that rarely makes it into the conversation: the ones you cannot see. No mobility aid, no visible sign. Just a person navigating a social world that was never designed with their brain in mind.
I know this personally. I spent time volunteering as a tutor for children on the autism spectrum, and it changed the way I see human communication forever. I watched a boy offer a brilliant, unconventional idea during a group project. The other kids didn't argue; they just shared a split-second, sidelong glance and a tight-lipped "Interesting, thanks." To him, it was an acceptance of his idea. To everyone else, it was a social execution. He spent the next week building on a proposal that had already been silently buried.
I remember another time, a girl was playing a board game with her best friend, a match that ended in a fluke victory for her. Her friend threw up her hands, laughing, and said: "Okay, that’s it—I’m never playing with you again." To the friend, it was the ultimate compliment; it meant, You’re so good it’s frustrating. She was grinning, her eyes crinkled in that joyful way that to her clearly showed her unseriousness. But for the girl, the grin was just a confusing mask. Her brain locked onto the word "Never." In an instant, the room went quiet, and her sobs became loud.
These situations are daily, repeated fractures in a person's ability to trust their own perception of the world. They affect friendships, school/work performance, family relationships, and mental health. And yet there is almost no technology built to help with them. Because these aren't simple "misunderstandings." They are data loss. In any other engineering field, we would build a bridge or a signal booster to fix a loss of data. Yet, in social interaction, we tell people to just "try harder" or "pay attention."
What struck me most, sitting with these students, was not that they were struggling — it was that they were struggling alone. The people around them, the ones who loved them, wanted to help bridge that gap. They just did not have a shared language for it. No tool. No way in. That gap — between someone who wants support and the people who want to give it — is exactly what Emotion Mirror is trying to close.
What it does
Emotion Mirror is a real-time communication support tool designed to be used openly, with full consent, among people who trust each other — a family at home, a person and their support worker, close friends navigating social situations together. The goal is not to train autistic people to perform neurotypicality or to mask who they are. The goal is simply to make visible the hidden layer of communication that neurotypical people navigate unconsciously — so that everyone in the room can understand each other more honestly. The Emotion Display uses the device's front camera and on-device machine learning to detect facial expressions and reflect the emotional tone of the room back to you — live, private, and instant. No data leaves your device. The Subtext Listener is the heart of the project. It listens through the microphone and analyses spoken language in real time, detecting eight categories of hidden social meaning:
Sarcasm — when someone says the opposite of what they mean ("Oh wow, great timing", "Bold choice", "Interesting outfit") Polite Refusal — when "maybe" means no ("We'll see", "I'll try to make it") Passive Aggression — when "fine" is not fine ("It's fine, I guess", "Do whatever you want") Backhanded Compliments — praise with a hidden sting ("Not bad for a beginner", "Surprisingly good") Veiled Frustration — the quiet exhaustion of repetition ("As I already said…") Social Pressure — when a choice is framed as the wrong one ("Everyone else is doing it") Exclusion Signals — the polite brush-off ("It's kind of a small gathering") Dismissal — when someone is not really listening ("Sure, sure. Right.")
When a signal is detected, a gentle alert appears on screen with a plain-language translation of what was likely meant — not as a verdict, but as a prompt for conversation. The result is something like having a trusted friend who can say: "Hey — I think that might have been sarcasm. Want to talk about it?"
How we built it
Emotion Mirror is a single-file progressive web app — no installation, no account, no server. It runs entirely in the browser. Emotion detection is powered by face-api.js, a JavaScript library that runs neural network models directly on the device. The camera feed is processed locally and never transmitted anywhere. Subtext detection is a custom two-tier pattern recognition engine built in JavaScript. Every phrase the microphone hears is run through two layers of analysis:
A high-confidence tier — a library of over 100 specific linguistic patterns that are almost never used without social subtext. A single match is enough to trigger an alert immediately. A medium-confidence tier — a broader set of weaker signals that are ambiguous on their own. Two or more must appear together before an alert fires. This is what prevents false positives on perfectly innocent sentences.
A critical technical challenge was that browser speech recognition never adds punctuation to transcribed text. This meant every pattern had to be written to work on raw, punctuation-free speech — something that broke the first version of the engine entirely and required a complete ground-up rewrite. The Web Speech API handles voice input and the Speech Synthesis API handles audio output — both are native browser features, requiring no third-party service and no network call. The entire app is deployable on a free static hosting platform with zero backend infrastructure, so that cost is never a barrier to access.
Challenges we ran into
Sarcasm is hard — even for humans. The entire premise of sarcasm is that the words themselves are innocent. "Interesting outfit" contains nothing grammatically suspicious. Detecting it requires understanding that certain words — interesting, bold, convenient, fascinating — carry a strong pattern of sarcastic use in specific constructions. Building a rule library broad enough to be useful, yet specific enough not to fire on genuine warmth, required careful, honest iteration. Speech-to-text strips all punctuation. Early versions of the detection engine relied on sentence-ending punctuation to help identify exclamations and rhetorical questions. When tested on real speech recognition output, it failed completely — because spoken transcripts arrive as flat, unpunctuated strings. Every single pattern had to be rewritten from scratch. The false positive problem. A tool that cries wolf is worse than no tool at all. If this app tells someone that a genuine, warm compliment is sarcasm, it does not just produce a wrong answer — it could damage a real relationship. The two-tier confidence system was built specifically to address this. Ambiguous signals only fire when they cluster together. Getting the ethics right. The most important challenge was not technical. It was asking the right question from the start: who is this for, and are they consenting to it? Thinking about this kind of tool, it is easy to drift toward a "hidden aid" framing — something invisible to others in the room. We deliberately rejected that. Consent is not a legal checkbox here; it is the entire foundation of what makes this tool safe and meaningful. Emotion Mirror is built to be used openly, by people who have agreed to use it together. Everyone in the room knows it is on. That transparency is not a limitation — it is the point. It transforms the tool from something used in secret into a shared resource that can actually open conversations, rather than just silently monitor them.
Accomplishments that we're proud of
We are proud of the privacy foundation. No audio is sent to a server. No transcripts are stored. No user account is created. For a person who may already feel exposed in social situations, the knowledge that this tool holds nothing about them — and never could — is not a feature. It is a prerequisite for trust. We are proud that it is accessible by design, not as an afterthought. Free to use. No installation. Works on a phone already in someone's pocket, opened like any other browser tab. And we are proud that it embodies the spirit of the EU Disability Strategy 2021–2030 — not just the letter of it. The Strategy calls for full social participation for people with disabilities. We believe you cannot have full social participation if the hidden rules of communication remain permanently illegible to you — and permanently unexamined by the people around you. Emotion Mirror asks both sides to show up differently.
What we learned
We learned that language is far less transparent than it appears. Neurotypical communication relies enormously on what is not said — on tone, implication, shared cultural scripts, and deliberate ambiguity. We treat this as normal. We do not teach it explicitly. We assume everyone just picks it up. People on the autism spectrum, people with social anxiety, people who grew up in different linguistic or cultural environments — they often do not "just pick it up." And we have decided, as a society, that this is their problem to solve. We learned that the right framing changes everything. A tool built around secrecy and passing produces a very different outcome — and a very different set of values — than a tool built around consent and shared understanding. The first asks an autistic person to hide that they need support. The second asks the people around them to be part of providing it. Those are not the same thing at all. We learned that building for accessibility forces better engineering. Every constraint — no punctuation in speech output, no assumptions about prior context, no tolerance for false positives — made the detection engine more disciplined and more honest about what it could and could not do. And we learned that the need for this tool is not niche. Social communication differences affect hundreds of millions of people worldwide — across autism, ADHD, anxiety, acquired brain injuries, and more. This is not a small problem with a small audience. It is a massive unmet need wearing an invisible sign.
What's next for Emotion Mirror — Feel the Room
The browser prototype proves the core technology works. The next step is building the hardware it was always meant to live in. The vision is a pair of lightweight everyday glasses with a discreet front-facing camera and a small LED indicator light on the frame — always on whenever the camera is active, visible to anyone nearby. The LED is not optional and it is not a concession. It is a design principle. It means that everyone in the room can see that the device is running. Consent is not assumed; it is made visible and ongoing. The glasses connect wirelessly to a paired earpiece that delivers gentle audio cues — a soft spoken word, heard only by the wearer, at the exact moment it matters. This form factor changes what the tool can do. A phone has to be held or propped and glanced at. Glasses are simply worn. They follow the natural line of sight through a conversation — a family dinner, a school corridor. They remove the friction of reaching for a screen at precisely the right second. And because the LED announces their presence, they carry the same honest, open character as any other visible accessibility aid. They are, by design, a visible accommodation — in the same spirit as a hearing aid or a communication board — and never a hidden recording device. On the software side, the next steps are: Multilingual support. Social subtext exists in every language, and the need for this tool is not limited to English speakers. The EU alone has 24 official languages, and the invisible disability community exists across all of them. Personalisation. Every person's social world is different. We want to let users — together with their family or support network — flag phrases from their own life that have caused confusion, building a personal dictionary of signals calibrated to their actual relationships and environment, not a generic model. Schools. A calm, shared tool that helps a student and their support worker understand a social moment together — without shame, without singling anyone out — could change the entire school experience of a child who currently navigates every interaction without a map. The EU Disability Strategy speaks of building a Union where every person, regardless of ability, can participate fully in society. We believe that full participation has to include the hidden conversation. The one that happens beneath the words, in the pause after "interesting," in the "fine" that means anything but. Emotion Mirror is a first step toward making that conversation visible — not just to the person who needs support, but to everyone in the room who wants to give it.
Log in or sign up for Devpost to join the conversation.