Blink Blink Blink Talkur

Talk, Text, or Call using only your Eyes!

Comment

Inspiration

My uncle had ALS, and he passed away this past month. Watching him lose the ability to move and then to speak was one of the hardest things I’ve been through. He understood everything around him, but by the end he could barely get a word out. The tools we found were expensive, depended on a camera and good lighting, stayed stuck on a screen by his bed, and couldn’t text his family or call for help when nobody was in the room. About 33,000 Americans live with ALS, and most of them will lose their voice. I built BlinkTalk for people like him, and for the families sitting next to them.

What it does

BlinkTalk lets someone who can’t speak or move talk, text and call for help using only their eyes.

Eyes in, no camera: three electrodes on the forehead read the eye’s electrical signal (EOG), so it works in the dark, lying down, and with drooping eyelids. A TI board recognizes each movement on the device itself: look left or right to move, two blinks to choose, three blinks to pause so they can rest, five blinks to call for help. Four blinks do nothing, so pausing can never be mistaken for an SOS. A screen made for eyes: one choice is lit at a time on an endless ribbon, with the message they’re answering on top and their words in large type. The colors are warm with no blue light, because it’s on at the bedside all night. Real iMessage: the patient has their own iMessage line through Photon. Family text them like anyone else, and photos show up described on screen. AI that never speaks for them: an NVIDIA Nemotron agent drafts four replies in their voice. Spell “mom tea” and it works out who it’s for and writes the full message. One double blink sends it. The dashboard counts the eye movements it saved. Help in five blinks: an SOS texts the first caregiver. If nobody replies “OK” within 45 seconds, it moves to the next person. When someone answers, the patient’s screen says help is on the way. Catch me up: a caregiver taps one button and Google Gemini writes a short handoff note: who the patient talked with, what they asked for, any SOS and how fast it was answered, and their heart rate. Family can be there from anywhere: a caregiver dashboard shows their words appearing letter by letter, a live eye trace, heart rate and the conversation. Almost fully offline: the eye model, the database, both screens and even Nemotron (through Ollama) can run on the laptop and board. Only texting and Gemini need the internet.

How we built it

Hardware: three forehead electrodes and a heart-rate (PPG) sensor feed a TI AFE4950EVM. A per-person scikit-learn model on a TI SK-AM62B (AM62 processor) classifies each eye movement on the device, offline, and calibrates for a new person in about a minute. SpacetimeDB is the entire backend: one TypeScript module with 12 tables and about 25 reducers holds every rule. That covers heart-rate alerts, phrase learning, message routing, and SOS escalation as a scheduled reducer, so it keeps escalating even if every screen crashes. The patient screen and dashboard are live subscriptions with no API layer. Photon 2.0: a Node hub polls the patient’s iMessage line, turns incoming texts and photos into database rows, and sends replies and SOS alerts from the same number. Replying “OK” acknowledges an alert. NVIDIA Nemotron: a tool-calling agent (get_conversation, get_patient_state, list_contacts, alert_caregivers, propose_replies, propose_messages), plus Nemotron’s vision model for photos. It runs on build.nvidia.com, or offline through Ollama. Google Gemini: gemini-flash-latest writes the caregiver briefings from the last few hours of conversation, alerts and vitals, with a plain summary as a fallback. Only first names and the patient’s own words are sent. Two models, two jobs: Nemotron for the patient, Gemini for the family. Front end: React + Vite for the patient screen and the dashboard. Testing: unit tests for the backend, agent, Photon bridge and speller, an end-to-end script against a live SpacetimeDB server, and browser tests that drive both screens at once.

Challenges we ran into

Making blinks a reliable language. People blink naturally all the time, so single blinks are ignored, the gestures are spaced out (2, 3, 5), and four is unused so pause and SOS can never be confused. AI that helps without speaking for someone. Nemotron only drafts, and the patient always chooses. Getting replies short enough to be worth a glance but different enough to be useful took a lot of prompt and tool design. iMessage rules. A shared Photon line can only reply to people who texted it first, so the hub remembers each caregiver’s conversation. Escalation that can’t silently fail. The SOS timer lives in the database as a scheduled reducer, not in a browser tab. Endless scrolling with eyes. The choice ribbon wraps forever without jumping, so the patient never loses their place.

Accomplishments that we're proud of

A full loop from an eye movement to a real iMessage on a family member’s phone and back. Five blinks reach a real phone, with escalation and a “help is on the way” confirmation for the patient. Measuring the AI instead of hand-waving: every AI-drafted message is compared with what spelling it would have cost. It runs almost entirely offline, with no camera. 60+ automated checks, including an end-to-end run against a live SpacetimeDB server.

What we learned

When the database is the backend, the clients get simple: the screens are just subscriptions. For assistive tech, the AI’s job is to save effort, not to talk. Every design decision got better once that was the rule. Accessibility is in the details: one thing in focus, big type, no blue light at night, and gestures that can’t be triggered by accident.

What's next for BlinkTalk

Testing with people living with ALS and their clinicians. Voice calls through Photon, with the patient’s replies spoken on a real phone call. A personal synthesized voice, and long-term tracking of gesture accuracy to help care teams notice changes. A single custom board in place of the two dev kits, to bring the cost far below today’s devices.

Built With

Share this project:

Updates

Submission history