Lumen

Dashboard

Inspiration

About 1 in 3 adults between 65 and 74, and nearly half of those older than 75, have hearing loss. Most of the house still assumes you can hear a beep.

A smoke alarm, a doorbell, someone calling your name from the kitchen: if you cannot hear it, or you are asleep in another room, that sound has to become a light on a phone you can actually read. Lumen is that light. Full access to the sounds of your own home is not a privilege. The tools should not only work one way.

What it does

Lumen turns spare phones into a home safety system for deaf, hard-of-hearing, and older people. Each phone is its own page. Together they turn sound and pose into captions and alerts you can see.

Four jobs, each on its own page:

Room nodes (/node). A spare phone listens and watches. It sends audio, body pose, and one still every few seconds. No live video leaves the phone.

Captions (/captions). Hold your phone up like a window. Speech appears in large, high-contrast type on the face whose mouth was moving, or at the bottom if the match is unsure. A line is tagged TO YOU when someone is facing you. Name calls, doorbells, glass, timers, and possible falls stay on screen until you tap them. Small gray text is how an alert gets missed, so the type stays big.

Bedside strobe (/strobe). When the kitchen hears a smoke alarm, the nightstand flashes and vibrates until someone taps it. You arm it on purpose. Nothing else on Lumen flashes fast.

Caregiver dashboard (/dashboard). The floor plan lights up the room an alert came from. Ask Lumen can answer “Where are my keys?” or “What happened while I was asleep?” It only reads memory. It never decides an alert.

Every alert and every tap is logged. A Databricks pipeline compares this week with last week for the same person. If she is slower to notice, or she misses more, the family gets a note that says worth a call. It never says diagnosis.

How we built it

Each page talks to one FastAPI brain over WebSockets. Face, pose, and object detection run on the phone with MediaPipe. On the demo rig, AST (sounds), Whisper (speech), and OWLv2 (what is in a still) run on an AMD Radeon cloud GPU. A Mac bridge plus cloudflared gives the phones HTTPS.

Sound direction

Three mics on the person, not one shared clock. The headset is the origin, the chest phone is about 30 cm away, the back camera about 28 cm behind. Training delays are calculated, not downloaded.

Sound travel time in seconds = distance in meters / 343.

At 343 m/s that chest gap is about 0.9 ms, roughly 14 samples at 16 kHz. A small model keeps a pick only if it is at least 55% sure. Otherwise we keep whichever mic heard the bump first. Direction never fires an alert.

Distance from the camera

A box is not a range. We use the camera’s focal length and a known object height: 1.7 m for a person, 1.5 m for a car.

Estimated distance = (focal length × known object height) / bounding box height.

Good to about plus or minus 30%. Enough for “car close, on your left.” A box cut off by the frame is treated as already inside 2.5 m.

Falls

Torso past 60 degrees from vertical, hips down more than 40% of standing height within 1 second, then 8 seconds of stillness. Leaving the frame means unknown, not fallen.

Smoke alarm

No model. Energy between 2.8 and 3.6 kHz. Three beeps in about 3 seconds fires the strobe.

History pipeline

Live alerts go out on the socket and do not wait on the warehouse. The same events are JSON lines, one file per hour. The Mac uploads new files. Auto Loader reads each file once.

Bronze keeps every raw line.

Silver drops bad rows: no alert kind, a timestamp before 2024, a test-button alert, blank captions, item sightings under 30% confidence.

Gold joins each alert to the first tap.

Time to notice = time of first tap minus time of alert.

No tap within 5 minutes counts as missed.

Worth a call only if the week before has at least 20 real alerts, and either median time to notice grew by more than 25% or misses at least doubled. One slow day does not page the family. The generated household is maria. The live one is home. They are never on the same chart.

Challenges we ran into

A strobe that waits on Databricks is a failed fire alarm, and a week-over-week number inside the alert loop is a lie until you have a week. The warehouse stays off the live path.

The three phones do not share a clock, so each half-second is lined up with the headset before we use the sound travel-time calculation. Distance from a box assumes you see the whole object, so we clamp it and we say plus or minus 30%. Free Edition has no personal access tokens, so the Mac sync uses an OAuth login. Test-button alerts are dropped in Silver so a demo click cannot look like a missed alarm.

Accomplishments that we're proud of

Live alerts and a caregiver history that share one log and do not share a latency budget.

A smoke-alarm path with no AI in it: a band check and a beep count.

Captions and alerts large enough to read at arm’s length at night. The Eclipse logo is the only color.

A family note that stays quiet until there are 20 real alerts in the week before.

Automated tests in CI on every pull request.

On generated maria data, misses go from 5 to 15 (2 to 7 at night) and the note says worth a call. That slowdown was planted to show the feature. It is not a real person. Real home logs can say what happened this afternoon. They are not long enough yet for a week-over-week percent.

What we learned

The hard part is not hearing the tone. It is who is asleep, who is in another room, and how big the words have to be.

One missed tap means little. A week of slower taps is a phone call.

The sound travel-time calculation is only as good as the mic positions, and the camera distance calculation gives only a rough range. We publish both limits.

What's next for Lumen

Ask “how’s she doing this week?” out loud, and still refuse to call a short log a trend. Test the walking view on real phones. Replace maria with a real household once the family agrees and the log is long enough for the 20-alert baseline.

Built with

FastAPI, React, Python, MediaPipe, Whisper, PyTorch, Databricks, WebSockets.