BlinkTalk

Inspiration

People with ALS, stroke or paralysis often keep a sharp mind but lose the ability to speak or move. Many rely on slow, expensive tools, and even a simple answer like "yes, I'm in pain" can take minutes. We wanted something that works on a normal laptop with no special hardware and lets people answer at the speed of a conversation.

What it does

BlinkTalk turns the camera on a laptop into an eye-controlled voice.

A caregiver asks a question out loud, for example "Are you in pain?"
BlinkTalk listens, and the AI suggests six likely replies on big, easy-to-see tiles.
The user looks left or right to move a highlight across the tiles.
A long blink picks a reply. The app speaks it aloud in a cloned voice (ours is a clone of a team member's voice, showing how a patient's own voice could be preserved). If the cloned voice fails, a default voice speaks instead.
A caregiver screen on a second device shows the conversation live.

It also has:

Emergency alert: five quick blinks in 2.5 seconds, or eyes closed for 2–4 seconds, start a 5-second countdown. If the user doesn't cancel, the caregiver's screen is alerted.
Rest mode: closing the eyes for more than 5 seconds pauses everything, so the user can rest without selecting anything by accident.
Bedside focus: the AI is built for a hospital or hospice. If someone asks something off-topic ("Who is Taylor Swift?"), the first tile becomes "Please ask something else" instead of a trivia answer.
Privacy: the camera video never leaves the user's device. Only text and eye-signal events go to the server.

How we built it

Eyes: the browser tracks the face with Google MediaPipe. A small classifier we trained from scratch on our own recorded sessions (4 people, 10 sessions, 2 with glasses) tells apart looking left, looking right, a natural blink and a deliberate long blink. Each user also calibrates once.
Understanding: the caregiver's speech is turned into text in the browser. Google Gemini reads it and suggests six replies. If Gemini is slow, the app falls back to safe default phrases, so it never freezes.
Voice: ElevenLabs speaks the chosen reply in a cloned voice. We cache every phrase and pre-generate the likely ones, so speech starts almost instantly. If the cloned voice fails, a default voice speaks instead.
Server: a small Python (FastAPI) service stores the conversation and sends live updates to the caregiver screen. It runs on a Vultr cloud server behind HTTPS.
Safety nets: every network call has a 5-second limit and a backup plan. The app keeps the server connection warm, and we have automatic deploys and a one-command rollback.

Challenges we ran into

Looking left was harder to detect than right. We re-recorded our training data and retrained the model.
Natural blinks vs deliberate ones. People blink about 15 times a minute. The app has to ignore those but catch a purposeful long blink and the emergency pattern.
Slow first connections on venue Wi-Fi. We saw 3-second stalls on a new connection, so every call has a 5-second limit.
The app hearing itself. The microphone picked up the app's own voice, so we mute listening while it speaks.
A team of four, one server, one weekend. Automatic deploys and rollback kept the live demo from breaking.

Accomplishments that we're proud of

A full loop that works end to end: a spoken question, an eye selection, then a spoken answer.
A real voice clone running in the loop, so a patient could hear their own voice again.
Privacy by design: no video leaves the device.
It runs in a normal browser, with no special hardware.
Eye control tested on people the model never saw: long blink 60/60, look right 48/48, look left 47/48. This includes glasses wearers.
Only 0.84 accidental selections per minute while a person is talking.
Reply tiles appear in about 1 second on the server (median 1.1 s over 20 live calls), and a chosen phrase starts speaking in under 0.1 s once cached.
The AI stays on bedside care: 12/12 off-topic questions were redirected and 12/12 bedside questions were answered normally.

What we learned

Reliability matters more than cleverness for assistive tools, so every feature needs a fallback.
Eye signals are noisy and different for every person, so calibration and real recorded data matter.
Small delays feel huge to someone waiting to be heard, so we cache and pre-generate voice.
How to ship a live service as a team: HTTPS, deploys, rollbacks and testing under pressure.

What's next for BlinkTalk

Better left and right detection, tuned per person.
Stronger emergency alerts, such as phone notifications that reach a caregiver whose phone is locked.
Personal phrase banks and a learned voice for each user.
Testing with real patients, caregivers and therapists.
More languages, and tablets mounted on wheelchairs.