Inspiration

Older adults want to keep living on their own, and the risks that threaten that independence are everyday ones: scam calls, food they can't safely read the label on, and falls. Numbers made it concrete: Americans over 60 lost $7.75B to internet crime in 2025, and 1 in 4 adults over 65 falls every year. Existing tools like phone apps or alarm pendants don't help in the moment, so we asked what a quiet helper in the wearer's ear could do. The goal was to protect independence, not take it away.

What it does

Ally runs on Mentra Live camera glasses. It listens through on-device speech recognition and whispers a short warning in the wearer's ear at the moment of risk, then sends the caregiver a short event, never footage. Four helpers are built:

  • Scam Shield hears things like "pay with gift cards right now" on a call and warns before money moves.
  • Food Check reads a package or a dish against the wearer's foods-to-avoid list and says "do not eat this" when it matters.
  • Ask and Calendar answer questions like "what's next on my schedule?" or "what does this letter say?", privately.
  • Fall check-in asks "Are you okay?" after "I've fallen", and alerts family if there's no answer.

How we built it

  • Phone side: a Mentra miniapp in TypeScript and Bun handles the wake word ("Hey Ally"), local speech-to-text, the camera and the whisper voice.
  • Server: a small Bun server calls Claude (Haiku for quick text decisions, Sonnet for reading photos) and keeps the schedule and foods list.
  • Safety by code: Claude only reads the words. Plain code makes every safety call, such as allergen matches, expiry dates and alerts.
  • Caregiver app: an installable web app shows live alerts with sound, plus an editable week schedule and foods-to-avoid list.

Challenges we ran into

  • Camera reliability: photos timed out or collided on the glasses, so we added a camera warm-up, retries and fallbacks.
  • No fall sensor access: the SDK's accelerometer isn't available on Mentra Live, so fall detection became a voice check-in plus an experimental loud-noise trigger.
  • Speech recognition mishears names and acronyms, so the wake word and scam phrases use fuzzy matching.
  • Keeping the AI honest: getting the model to read labels verbatim, and never decide safety on its own, took many rounds.

Accomplishments that we're proud of

  • Privacy by design: events, not footage. Audio never leaves the phone, there's no face recognition, and the camera only runs when the wearer asks and announces itself.
  • A clear safety rule: the AI reads, plain code decides. A pizza slice with cheese is "do not eat this" for a dairy allergy, even when no label is readable.
  • A full loop: one demo takes a spoken or visual trigger through to a whisper in the wearer's ear and an alert on the caregiver's phone.
  • Respect for dignity: the wearer picks the wake word, and everyday requests stay private.

What we learned

  • Trust comes from restraint. Wearers and families trust a system more when it says what it can't do, like "check the label" when it can't read one.
  • Hardware shapes the product. Camera latency and sensor access decided what we could promise, so we designed around what the glasses can actually do.
  • The AI should do the reading, not the deciding. Pairing a model with simple rules made it both safer and easier to test.

What's next for Ally

  • Real fall detection: once the glasses expose motion sensors to apps, detect falls directly instead of relying on voice.
  • More food and medicine checks: barcode lookup and leftover age guidance, plus reading prescription labels.
  • Reliable alerts: real phone push notifications and an acknowledgement button, so "help is on the way" is true.
  • Privacy upgrades: blur faces on the device before any photo is uploaded, and support more languages and voices.

Built With

Share this project:

Updates

Submission history