Glance

Hands-free communication — reply with a glance and a blink.

Comment

Glance

https://x.com/KaiK10676/status/2069794456371605568

Inspiration

My grandma is bedridden right now after Whipple surgery. She can't pick up a phone or tap a screen with her hands. When I visit or call, I see how hard it is for her to do something that should be simple: hear from family and answer back. Commercial AAC devices cost thousands of dollars, need clinical setup, and still don't feel like a normal conversation with the people who love her.

That gap became Glance. We wanted communication that works on a laptop she already has — webcam, microphone, browser — with no hands, no keyboard, no mouse, ever on the patient side. If she can look and blink, she should be able to reply. If she can't use the camera, she should still be able to call for help.

What it does

Glance is a hands-free AAC platform for motor-impaired patients and their families.

For patients (like my grandma):

  • Receive messages from family on a large, high-contrast screen
  • Hear messages read aloud in a cloned voice — so it sounds like Mom or Dad, not a robot
  • Reply using gaze (look at a target and dwell) or blink when the camera isn't available
  • Answer yes/no questions with giant on-screen targets ("look up = yes", "look down = no")
  • Trigger SOS by sustained vocalization (microphone) or gaze — even when the camera is off
  • See an animated Blob character that reads messages, shows tone, and guides where to look

For caregivers (family on the dashboard):

  • Compose and send messages from a simple web dashboard
  • Clone their voice once; messages are delivered in that voice with tone-aware TTS
  • Manage multiple patients, see online/offline status in real time
  • Get instant alerts for SOS, replies, and help requests
  • Control camera schedules so the webcam only runs when privacy and battery allow

How we built it

We used a pnpm monorepo with Turborepo:

Piece Stack
Patient app Next.js 15, TypeScript, Tailwind, Framer Motion
Family dashboard Next.js 15, Better Auth, Radix UI
Real-time layer Standalone socket.io server for live messages and alerts
Database Postgres 16 + Drizzle ORM
Gaze MediaPipe Face Landmarker — iris position → direction (up/down/left/right/center)
Voice ElevenLabs instant voice cloning + TTS
AI (bounded) LLM tone classification, yes/no detection, ranking curated reply phrases — never auto-sending content the patient didn't select

Architecture choices that mattered:

  1. SelectEvent — one event whether input came from gaze-dwell, blink, or scan-mode; UI doesn't care which modality fired.
  2. ScanMode fallback — if the camera is off, denied, or no face is detected, targets auto-highlight on a timer; blink confirms.
  3. CameraWindow — camera tracks only run inside scheduled windows; every activation has a paired track.stop() on all exit paths.
  4. Hard constraints in CI — static guards and tests enforce zero-hands patient UI, camera lifecycle, AI content gate, and SOS-without-camera.

We built in phases: message backbone → gaze + voice + SOS → multi-patient dashboard → production hardening and a feedback pass (read receipts, offline queue, send-as personas, media upload).

Challenges we ran into

Gaze is harder than it looks. Webcam gaze isn't a screen cursor — it's directional classification from iris landmarks, smoothed over many frames. Lighting, head angle, and recovery fatigue (especially post-surgery) all affect accuracy. We added ScanMode so the system degrades gracefully instead of failing silently.

Browser security vs. zero hands. Browsers block autoplay and microphone access without a user gesture. We needed a one-time caregiver "Start session" tap to unlock AudioContext — a compromise that still keeps day-to-day patient use hands-free.

Camera privacy and battery. Leaving a MediaStream running is wrong for trust and hardware. CameraWindow scheduling plus strict lifecycle rules meant every code path that opens the camera had to close it — unmount, schedule end, permission denied, navigation.

AI without stealing agency. Families want help composing; patients must never feel spoken for. We limited AI to tone classification, yes/no detection, and ranking explicit phrase choices — with a blink-confirm before anything is sent.

Real-time at scale. Message delivery, SOS, and presence needed sub-second paths. Splitting a dedicated WebSocket server from Next.js API routes, with room-based subscriptions per patient, kept the patient UI responsive without blocking serverless handlers.

Testing hands-free UX. We can't fully validate "95% first-dwell accuracy" in CI. We invested in hard-constraint regression tests, Vitest API coverage, and pure modules for SOS threshold logic — while marking real-device validation as the next gate.

Accomplishments that we're proud of

  • A complete hands-free patient loop: receive → hear in family voice → gaze/blink reply → caregiver sees it live
  • Dual-path SOS that works without the camera — critical for someone bedridden who may not have the webcam on
  • Voice cloning wired end-to-end: record on dashboard, clone via ElevenLabs, play on patient with tone-aware delivery
  • Multi-patient family dashboard with scoped access, personas ("Mom", "Dad"), and real-time unseen badges
  • Four hard product constraints encoded in repo policy and tests — not just documented in a PRD
  • Offline resilience: pending message replay on reconnect so nothing sent while the device was off is lost

Most of all: we built something aimed at a person we love — not a generic "accessibility demo."

What we learned

  • Accessibility is architecture, not a CSS pass. Input modality, camera lifecycle, and audio unlock had to be designed in from day one.
  • Fallback paths are features. ScanMode isn't "worse gaze" — it's how Glance stays usable when the camera is off, denied, or MediaPipe is still loading a 15–30MB model.
  • Voice changes emotional distance. Hearing a cloned family voice on a hospital bed hits differently than system TTS. Worth the ElevenLabs integration complexity.
  • Constraints make better software. "No auto-filled messages" and "SOS without camera" forced cleaner boundaries between AI assist and patient intent.
  • Monorepo paid off. Shared design tokens, schema, and SelectEvent types kept patient and dashboard aligned as modalities grew.

What's next for Glance

  1. Real-device validation with patients in recovery — including my grandma — tuning dwell times, ScanMode speed, and SOS thresholds for weak vocalization
  2. Clinical pilot with a small cohort (post-surgical, ALS, locked-in) and measured gaze accuracy on hardware families already own
  3. Compose assist for caregivers — templates and phrasing help on the dashboard, still separate from patient-side autonomy
  4. Docker / one-command deploy so a family can run patient + dashboard + ws-server without wrestling with env vars
  5. Pitch and partnerships — hospitals, rehab units, and AAC nonprofits; position Glance as affordable, family-first messaging on existing devices

Glance started when we realized my grandma couldn't operate a phone with her hands. We're building so she — and anyone like her — can stay connected, answer back, and call for help using only her eyes, her blink, and her voice.

Built With

Share this project:

Updates