Headvoice

HeadVoice turns any webcam into a voice: tilt your head to choose, hold a blink to say it.

Comment

Inspiration

Some people can't speak clearly and can't use their hands, but they can still move their head a little and blink. The tools built for them are often slow, expensive, or need special hardware. Meanwhile almost every home already has a webcam. We kept coming back to one small moment: someone wanting to say "I need water" and having to wait for someone to guess. We wanted to see how far an ordinary laptop camera and OpenCV could go toward fixing that.

What it does

HeadVoice turns a webcam into a voice.

  • Tilt your head to highlight a message.
  • Hold a long blink to confirm it, and the computer says it out loud.
  • Two quick blinks cancel. Nothing is ever spoken without your own confirming blink.

A short calibration (about 25 seconds) measures how far you can comfortably tilt in each direction compared with your own natural jitter, and how you blink. From that it picks the layout that suits you: a four-way menu if you move well in all directions, or left and right scanning if up and down aren't reliable. It also sets your personal thresholds.

While you use it, an agent on Amazon Bedrock watches the numbers coming out of OpenCV (how big your tilts are compared with your range, how many near misses, signal quality) and makes small, reversible changes. If you're getting tired and your tilts are shrinking, it makes them easier to reach. If the lighting is bad, it asks you to hold still instead of changing your screen. A helper can add the person's own phrases, like "Turn on the cricket", and they stay on that computer.

The camera picture never leaves the computer. Only numbers go to the cloud.

How we built it

OpenCV 5 does all the seeing

Everything visual runs on the user's computer with OpenCV 5.0.0 (opencv-contrib-python==5.0.0.93).

  • Face detection with cv2.FaceDetectorYN and the 2026 YuNet model. OpenCV 5's new ONNX Runtime engine needs YuNet's dynamic-shape export, so we had to move off the older model.
  • 68 facial landmarks with the LBF facemark model from cv2.face.
  • Head pose with cv2.solvePnP for pitch and roll. For yaw we use the nose position along the eye line, because PnP with a generic face model kept flipping between two answers.
  • Eye state with our own small classifier: cv2.HOGDescriptor features on aligned eye crops, fed to a linear cv2.ml.SVM. We built this because the landmarks around the eyelids barely move when you close your eyes, so the usual eye aspect ratio trick was basically a coin flip for us. The classifier runs in 0.17 ms per frame.
  • Staying robust: cv2.createCLAHE for dim faces, cv2.calcOpticalFlowPyrLK as a short fallback when tracking drops, a One Euro filter for smoothing, and a check that catches webcam drivers that freeze on one frame.

The whole pipeline runs at the camera's 30 fps on a normal laptop CPU. Detection takes about 10 ms, landmarks 4.5 ms and pose 0.6 ms.

We use the same OpenCV 5 code in the cloud too. The Lambda image ships the headless build of OpenCV 5, so a calibration clip (sent only with explicit consent) is analyzed by exactly the same code as on the device.

AWS runs the agents and the backend

It's all one CDK stack in us-east-1:

  • Amazon Bedrock with Claude Haiku 4.5 using Converse tool use, behind a Bedrock guardrail (content filters, prompt attack filter, PII anonymization, and a blocked topic for medical advice).
  • 9 Lambda functions from a single container image that holds OpenCV 5 and our package.
  • API Gateway HTTP API with a Lambda token authorizer and throttling.
  • S3 encrypted with a customer-managed KMS key. Calibration clips are deleted as soon as they're read.
  • DynamoDB as an audit table for every agent decision.
  • CloudWatch logs, metrics and a dashboard.
  • CloudFront for the website and the Windows download.

Each agent decision takes about 3.5 seconds end to end and costs about $0.006. All 30 real agent decisions so far cost $0.17 in total. scripts/smoke_cloud.py runs 9 end-to-end checks against the live stack, including checking that a bad token gets a 403, and all 9 pass.

The agent is on a short leash

The agent can look up more evidence first, then make one change: adjust sensitivity, adjust timing, switch layout, ask the person to hold still, suggest recalibration, or do nothing. The device checks every request against fixed limits before applying it. It keeps a one-step undo and measures what actually happened 60 seconds later. The agent can't speak, can't send alerts and never sees video. The decision about what gets said is plain, tested code on the device, covered by 11 safety rules and 125 automated tests.

Real traceability

This is the part we're proudest of. Every agent decision leaves a full trail:

  1. Why it fired: the trigger, like a quality drop or a run of near misses.
  2. What it saw: the exact numbers it was given.
  3. What it checked: any extra evidence it asked for before deciding.
  4. What it did, and why, in its own words.
  5. Whether the device accepted it, and if not, why.
  6. What happened next: the measured outcome a minute later.

These go to the local session log and the DynamoDB audit table. On top of that, our 13 real sessions are published (anonymized, with landmarks and face boxes stripped) on the judge replay page. Anyone can pick a session, replay it through any interface setup, and read every decision the agent made.

One real example from a 7.9-minute session: at 60 seconds the agent lowered sensitivity, pointing out that tilts were reaching only 0.54 and 0.46 of the calibrated range and that 21 of 34 attempts fell short. At 120 seconds, with "down" still failing, it switched the layout. At 170 seconds the signal quality dropped and it asked for the person to hold still. In a different session it asked for a sensitivity of 0.4, which is outside the allowed range, and the device refused it. All of that is in the logs, with numbers, not reconstructed after the fact.

Evals

We were careful to keep each kind of evidence separate and labeled for what it is.

Real sessions (n=1). 13 live sessions by the developer, 22.5 minutes, 30 real Bedrock decisions. Face found in 98.8% of frames. When we replayed them, the blink-confirmed setups selected nothing in 100% of the low-signal windows, while dwell selection (no blink) selected something in 6% of them. That result is why we kept blink confirmation.

Simulated users study (simulation, not people). 30 simulated users (typical, limited up and down movement, and tremor-like noise) with settings based on the developer's measurements. Each one drives the real app in a closed loop, so it reacts when the layout changes, and setup E calls the real deployed agent.

Task success Fixed menu Dwell Calibrated + Rules + Agent
Normal 0.65 0.65 0.93 0.92 0.93
Simulated fatigue 0.23 0.23 0.59 0.75 0.78

Calibration made the biggest difference: limited-range users went from 49% to 100% task success. Under fatigue, adaptation won back a lot of the loss. The agent did slightly better than simple rules for most users, but worse for tremor-like users, and we report that too.

Component evals on real data. The eye-state model caught 24 of 24 held-out long blinks across three recordings (with and without glasses). For the message suggestion agent we ran 40 scenarios on real Bedrock calls. The evidence-first prompt abstained correctly on 93% of ambiguous cases, against 43% for a careless prompt, and produced no invalid outputs.

Challenges we ran into

  • The face detector loses tilted faces. YuNet stops finding a face at about 20 degrees of roll, which is exactly the gesture we need. We fixed it by carrying a box forward from the last good landmarks and refitting. On our first live session that took tracking drops from 109 down to 9.
  • Eyelid landmarks don't move. We spent a while tuning eye aspect ratio before admitting it just didn't work on our webcam. Training our own HOG and SVM classifier solved it.
  • Head pose flipping. PnP yaw jumped between two solutions 155 times in one session. Switching to simple 2D geometry for yaw cut false "turned away" frames from 14.3% to 2.5%.
  • A phone webcam app that froze and kept sending the same frame. The app looked dead. Now we detect it and the camera picker marks frozen cameras.
  • The agent misread fatigue as bad signal. Our first prompt answered a clear fatigue window with "hold still" 3 out of 3 times. We rewrote the prompt so it tells signal problems apart from shrinking movement, then checked it again with real calls.
  • The agent acted too early. Live, it switched the layout 33 seconds into a session based on glances it counted as tilts. We added a one-minute warm-up, a stricter definition of a deliberate attempt, sensitivity changes before layout changes, and a helper layout lock.
  • API Gateway's 30 second limit was too short for agent replays of long sessions, so the judge page serves precomputed replays from S3. A Bedrock outage taught us never to cache a failed replay.

Accomplishments that we're proud of

  • A full OpenCV 5 pipeline that runs in real time on a normal laptop, including an eye-state classifier we built ourselves.
  • An agent whose every decision can be traced from trigger to measured outcome, with the device acting as the final check.
  • Being honest in the report. We list 12 failure cases, and we keep real data and simulation clearly apart.
  • A product that feels like a product: a landing page, a four-step setup, a live camera window, a page for your own messages, and a one-click Windows launcher.

What we learned

  • The classic tricks don't always hold up on a cheap webcam. Measuring first saved us from tuning things that were never going to work.
  • An LLM agent is much more useful when it's given numbers, a small set of reversible actions, and its own past results. Giving it more freedom made it worse.
  • Logging the outcome of every decision mattered more than we expected. Most of our agent fixes came from reading real traces, not from guessing.
  • Fail closed is the right default. When the signal is bad, doing nothing beats doing the wrong thing.

What's next for Headvoice

  • Testing with people who actually rely on tools like this, and with their care teams. That's the most important next step, and nothing replaces it.
  • Fixing the agent's known problems: it can still flip layouts back and forth, and it sometimes changes settings when it should only ask the person to hold still. The plan is a device-side rule for quality drops and a minimum gap between layout changes.
  • Retraining the eye-state model on more people, and handling eyes looking down.
  • More languages and voices, and support for Mac and Linux.

HeadVoice is a research prototype, not a medical device.

Share this project:

Updates

Submission history