HeadsUp!

Inspiration

For blind and low-vision adults, a white cane or guide dog handles the ground well—but branches, signs, and truck mirrors sit at head height, where canes do not reach. In a survey of ~300 blind travelers, head-level collisions were common; tree branches were the leading outdoor cause. At the same time, independence is not only navigation: reading a walk signal, finding keys, sending an email, or letting a sibling know you are okay still often means asking someone else or juggling a phone.

We built HeadsUp for people like Maya—working, commuting, proud of knowing every curb on her route—who want one voice-first interface on glasses they already wear, not another device to manage. We position HeadsUp alongside the cane: it adds an eye-level camera and memory; it does not replace touch, drop-offs, or ground texture.

What it does

HeadsUp is a voice-only assistant on Meta Ray-Ban glasses. You speak; the system listens, looks, remembers, and answers out loud through the glasses.

Hazards & describe: A local detector flags people, bikes, cars, and obstacles with short spoken warnings. On request, Claude reads the scene and pedestrian signals—it never says a crossing is “safe.”
Memory: Frames are captioned and stored in SpacetimeDB; Moss powers fast semantic recall (“Where did I put my keys?” → time, place, and a thumbnail on the family dashboard).
Act: Voice goals route to a browser agent; drafts are read back and require explicit confirmation before send.
Family dashboard: A trusted contact sees live alerts and recall snapshots and can step in when the user says they are lost.

One utterance is routed to the right mode (describe, recall, browser, alert, etc.) so we could ship a tight demo loop instead of four fragile separate apps.

How we built it

Pipeline: Meta Wearables Device Access Toolkit (glasses camera + mic) → phone relay → Python/FastAPI backend over WebSocket → mode handlers → Cartesia Sonic TTS back to the glasses. Laptop audio and transcript mirror the session for judges (open-ear speakers are quiet in a noisy judging hall).

Stack highlights:

Voice: Cartesia Ink 2 (STT + turn detection) and Sonic 3.5 (streaming TTS)—the UI is entirely spoken.
Routing: TypeSafe Jev for fast intent classification; low-confidence turns fall through to Claude Sonnet.
Vision: Ultralytics YOLO (nano) for always-on hazard hints; Claude Haiku for descriptions and sign reading.
Memory: SpacetimeDB timeline + Moss hybrid search; React dashboard with live subscriptions.
Browser: Jev Ultrafast on Chrome for approved actions from voice.

We also built a three.js pitch deck (headsup-pitch in the repo) that tells the problem story in ~60 seconds before the live demo.

Challenges we ran into

Hardware & venue: Glasses streaming over Bluetooth is sensitive to bandwidth and pairing; we built on MockDeviceKit early and kept a webcam/phone-camera fallback on the same WebSocket so the backend never depended on a single device.
Latency & noise: 8 kHz glasses mic and congested Wi‑Fi pushed us toward short command phrases, small JPEGs, ethernet on the backend laptop, and pre-generated hazard phrases where we could.
Scope: Four modes in 39 hours was unrealistic; we froze on voice loop + hazards + memory (P0) and treated browser control as P1 with a tested fallback site.
Trust: Judges and sponsor tracks care about overclaiming safety—we document failure modes, store only user-named people, and refuse “safe to cross” wording.

Accomplishments that we're proud of

End-to-end voice loop on glasses (or fallback camera) with routing to multiple modes from natural speech.
A demo that feels like a story: ground vs. air hazard gap → branch collision → rewind with HeadsUp → live chair / crosswalk / keys / family alert.
Memory recall with time, place, and dashboard thumbnail in one breath.
Hitting five sponsor-shaped integrations (Meta glasses + connection story, Anthropic vision/reasoning, Cartesia-native UI, Moss search, SpacetimeDB backend) without treating any of them as bolt-on README filler.
Shipping a single-file offline pitch for unreliable venue Wi‑Fi.

What we learned

Eye-level assistive tech is as much about what you do not promise (curbs, “safe”) as what you detect.
Voice-first UX forces ruthless brevity: two-sentence replies, confirm-before-send, and a visible transcript for bystanders and judges.
Contracts between subsystems (mocked WebSocket payloads, stubbed /recall, /ingest) let four engineers work in parallel without blocking on glasses pairing at midnight.
Real users (and blind/low-vision mentors we talked to at the event) care less about model names and more about latency, reliability, and dignity—not being the person who always asks strangers for help.

What's next for HeadsUp!

On-device vision for more hazard classes and lower cloud dependency; optional local inference when hardware is available.
Richer work mode (order checks, label reading) once P0 latency is stable.
Privacy controls: one-tap delete, retention limits, and clearer policies for bystanders in captions only (no stranger ID).
Production path: push-to-talk vs. VAD tuning, iOS/Android parity on DAT, and pilot with blind and low-vision adults who commute—not lab demos only.
Family dashboard: two-way voice from a trusted contact into the room speaker, with explicit user consent.

Team: Rishit S, Sharif A, Kevin T, Marvin Z · Cal Hacks 13.0