TacMask

Inspiration

TacMask is meant to add awareness, not to "fix" anything.

What it does

TacMask is a wearable with a Raspberry Pi and a webcam, plus an iPhone app. The phone is the interface and makes all the internet calls. The Pi only does local work.

Deaf mode

Sound alerts: The Pi classifies sounds with Google's YAMNet model and sends candidate sounds to the phone. Gemini decides whether the wearer should be alerted, then the app shows a banner and plays a haptic pattern that is different for each kind of sound. Visual hazards: A local YOLO model checks webcam frames for things like dogs or people. Gemini decides whether a detection is worth an alert. Live captions: When the Pi hears conversation nearby, the phone captures speech, sends it to ElevenLabs for transcription, and groups the text into lines. Quick replies: Tap Reply on a caption and Gemini suggests three short replies. Tapping one speaks it aloud. Typing your own reply is always available. Fall check-in: After a fall, the phone asks "Are you okay?" with a 30-second countdown. If the wearer doesn't answer, a Call button opens the dialer with an emergency contact.

Blind mode

Describe: One button asks the Pi for a photo. The phone sends it to Gemini, which describes what's in front of the wearer, and the app speaks the result. Results work with VoiceOver, and there's a repeat button. How we built it Phone: SwiftUI and CoreBluetooth, with Core Haptics for alert patterns. Pi: Python, with YAMNet and YOLO running locally. The Pi never connects to the internet. Communication: Bluetooth Low Energy. We chose it over WiFi because it doesn't need a hotspot and holds up better when the phone is in the background. AI: Gemini for alert triage, reply suggestions, and photo descriptions. The app falls back to the raw sound label if Gemini is unreachable, so it never goes silent. Speech: ElevenLabs for captions and spoken replies. Blind mode uses the phone's built-in voice, or VoiceOver when it's on. Secrets: API keys live in a git-ignored file on the phone and are never committed. Challenges Scanning: A generic BLE scan in a crowded room can take minutes. We filter by service UUID so the phone finds the wearable quickly. Haptics: The haptic player had to be kept alive while it played, or iOS cut it off immediately. Fixing that took a while to track down. Unknown labels: Gemini labels are free-form, so haptics match by keyword with a default pattern for anything unmatched. Fall calls: iOS apps can't place phone calls automatically. We first tried Twilio, but trial accounts blocked the call. The app now uses a one-tap dialer instead. Camera limits: The wearer can't see what the camera points at, so a bad angle can produce a bad description. The prompt tells Gemini to say when the view looks like ground or a wall. Pi stability: The fall sensor's USB library crashed the Pi server during testing. The fix is deferred. Accomplishments A complete phone app that builds for iPhone, with alerts, captions, replies, speech, fall check-in, and blind mode. Sound and hazard alerts passed through Gemini to the phone, with the fallback behavior working. Blind mode follows VoiceOver conventions instead of adding its own narration layer. What we learned Put the internet calls on the phone. The Pi stays simple, and it still works when there's no signal. Check whether a platform can do something before designing around it. iOS can't place calls on its own. Sending small text messages over Bluetooth works better than streaming audio or video. What's next Calls: Automatic emergency calls through a paid Twilio account, with the emergency contact verified and a server holding the credentials. Blind mode: Testing with blind VoiceOver users, plus a way to aim the camera. Fall sensor: Process isolation for the sensor code, so a crash can't take down the whole Pi server. Background operation: Keeping alerts running when the phone is locked.

Before submitting, check these, since I can't verify them from the code:

Whether the team has run the sound, hazard, and fall paths end to end against the Pi. The "Accomplishments" section assumes they have. Whether ElevenLabs text-to-speech works on your plan. It needs a paid plan, so captions and replies may not speak. Whether the Gemini model gemini-3.1-flash-lite accepts photos as well as text.