sense

Homepage
Hearing Page
Sight Page
Care Page

Inspiration

I kept thinking about a simple problem. The world produces information constantly, but not everyone can receive it the same way. A doorbell rings. A smoke alarm goes off. Someone calls your name across a crowded room. For most people these are forgettable moments. For 1.5 billion people with hearing loss, they're missed entirely. For someone with low vision navigating an unfamiliar space, the chair three feet ahead is invisible until it isn't. For a senior living alone, a fall at 2 a.m with no one to help isn't a hypothetical. It's a Tuesday. What hit us wasn't that these problems were unsolved. It was that they were almost solved. The information already exists. A microphone can hear a smoke alarm. A camera can see a chair. A phone can vibrate. The gap wasn't hardware. It was translation. So we asked a question. What if information didn't have to arrive through a specific sense? What if we rerouted it?

What it does

Sense is one app built for three communities that the world has consistently underserved. Hearing: For deaf and hard-of-hearing users. Sense listens constantly using YAMNet, a model that knows 521 sound classes. When it detects something that matters (a smoke alarm, siren, doorbell, baby crying, glass breaking) it triggers a fullscreen flash. It alerts you when someone says your name. And it lets you teach it any sound in five seconds, storing a personalized audio embedding compared against background noise using cosine similarity. Sight: For people with low vision. A GPU accelerated magnifier zooms up to 8×. Freeze mode holds the frame still for shaky hands. Filters include high contrast, inverted, and yellow on black. A text reader zooms into exactly what you're looking at and reads it aloud. Object detection draws warnings around people, chairs, and cars — with a red flash if something is approaching. "2 people ahead. Chair on your right." Care: For older adults living alone. Full-screen medication reminders with Taken / Snooze / Skip and timestamps. A voice interface that understands plain language — "remind me about my dentist appointment tomorrow at 2:30." A Watch Over Me mode that listens for a thud, sounds of pain, or the word "help," then shows "Are you OK?" with a countdown. One-tap emergency contact with your location. Alert sharing so a caregiver gets notified about a fall, missed medicine, or smoke alarm automatically. A food and label reader that extracts calories, allergens, and expiry dates from a photo — and says "Not captured" instead of guessing. Everything runs on the device. No cloud APIs at runtime. No account. No data leaves the phone.

How we built it

We used four AI models that run only in the browser. YAMNet for sound, COCO-SSD for objects, Tesseract for text, and Chrome Speech for voice. No backend at runtime. Then we built everything on top. The part we're probably most proud of is the alert engine. We realized quickly that raw model output was way too noisy to be of any use. So we built a 2 of 3 confirmation system where a sound has to be detected across multiple frames before it actually alerts. We added per class confidence thresholds, cooldown windows, and loudness trend tracking so it knows if a siren is getting closer. We also built custom sound learning from scratch. You record five seconds of a sound, we store a compact audio embedding locally, and future sounds get matched against it using cosine similarity. The whole thing runs offline. We used Claude and Claude Code to help build it, but no AI API touches the app at runtime.

Challenges we ran into

Getting the model to detect a doorbell was honestly the easy part. Getting it to not go off every second in a room was where we spent our time. The thing we kept coming back to was trust. If Sense alerts when nothing happened, a deaf user might start ignoring every alert after that. So false positives weren't just an accuracy problem, more of a product problem. We tuned the confirmation engine more than anything else in the project. Custom sound learning also gave us more trouble than expected. Cosine similarity between embeddings sounds elegant until you're dealing with three different microphones, a loud HVAC system, and a room full of people talking. We kept the threshold adjustable for exactly this reason. And then there's the thing we genuinely couldn't fix: it's a web app, so it can't run in the background. Screen off means no alerts. We put it on our honest limits slide. We're not proud of it but we're not hiding it either.## Accomplishments that we're proud of 58 tests passing. Zero cloud APIs at runtime. A label reader that says "Not captured" instead of imagining a nutritional value. But honestly the thing we're most proud of is the honest limits. We stood in front of judges and listed every gap in what we built before anyone asked. That felt like the right thing to do and also terrifying. We think it was the right call. We also just... actually care about this. Every small decision — the Snooze button on the medication reminder, the freeze frame for shaky hands, the "Not captured" label — came from thinking about a specific person who would use this. Not a user persona. An actual person in an actual situation. That kept us honest throughout the whole weekend.## What we learned Scope is a feature. We had to cut things we liked to protect the three things that actually mattered. A model that's right 90% of the time is useless if the other 10% destroys someone's trust in the whole system. Reliability matters more than accuracy. On device isn't just a technical decision, it's a values one too. We didn't want audio and camera data leaving anyone's phone and honestly once we framed it that way it made every other architecture choice easier. We also learned that saying "here's what we can't do yet" is better than pretending you built something you didn't.

What's next for Sense

Native app so alerts work with the screen off and SOS texts can send automatically. Real accuracy with actual people from the communities we built this for. Live health device connections. Honestly, just more time with real people telling us what we got wrong. We think there's something real here. We want to find out.