Inspiration
In a stroke, time is brain: roughly 1.9 million neurons are lost every minute a large-vessel stroke goes untreated. Clot-busting drugs only work within about 4.5 hours of when the patient was last known well, and clot removal within 24 hours, so when a stroke is noticed decides what treatment is still possible.
In the hospital, that noticing happens through periodic neuro checks. After a stroke treatment, nurses check every 15 minutes for 2 hours, then every 30 minutes, then hourly. A stroke that starts between checks, or overnight in a sleeping patient, can go unseen until the next one, and then nobody knows exactly when it started.
We asked: what if every bed had an extra set of eyes that never blinks? That's Eye-CU, a pun on ICU.
What it does
Eye-CU monitors many beds at once from ordinary cameras (laptop webcams, iPhones on a stand, recorded clips) and covers the face, eye and arm parts of the BE-FAST stroke screen (Balance, Eyes, Face, Arms, Speech, Time):
- F (face): it learns each patient's own normal face, then alerts on a new, lasting one-sided droop, sensitive to about 2.5 mm of change at the mouth corner.
- E (eyes): it notices when the eyes stay fixed to one side, and checks whether they can move fully left and right.
- A (arms): a voice-guided check asks the patient to close their eyes and hold both arms out, then measures arm drift.
- T (time): every alert records an onset time and shows countdowns for the two treatment windows.
A nurse-facing ward dashboard (Azure Medical Center, Unit 4B) shows every bed's live video, a 10-minute trend, alerts with a code-stroke checklist (team paged, glucose, NIHSS, CT/CTA, neurology, decision), an activity log stamped with staff initials, and a hedged localization hint ("may be a right hemispheric stroke").
How we built it
Face. MediaPipe's Face Landmarker gives 478 3D landmarks per frame. We measure mouth-corner, eye-opening and brow asymmetry, corrected for head pose. Each feature becomes a robust z-score against the patient's baseline:
$$z = \frac{x - \operatorname{median}(x_{\text{baseline}})}{1.4826\,\operatorname{MAD}(x_{\text{baseline}})}$$
These are weighted, smoothed over time, and alert once they stay above threshold. Frames where the patient is talking, blinking or turning their head are skipped, not guessed at.
A learned second opinion. We trained four architectures:
- TCN: a temporal convolutional network on face landmarks.
- 3D-CNN: convolutions over the face surface, built from landmarks.
- Graph CNN: a spatio-temporal graph network over the landmark mesh.
- ViViT: a video vision transformer on the raw video frames.
The three landmark models saw 2-second windows of frontalised 3D face landmarks (152 points at 10 fps), with per-point asymmetry and motion features. All four were trained on the CC-BY-4.0 palsynet facial-palsy dataset (49 videos). The landmark models were also trained on extra videos of healthy people on webcams and making acted expressions, so that a big smile isn't read as palsy. Every model was evaluated with subject-level 5-fold cross-validation, so no person appears in both training and testing.
We chose based on two things: validation results, and how well each model generalized to new footage. For the second, we ran the models on our own webcam and phone clips with a synthetic one-sided droop. ViViT did poorly, likely because learning straight from pixels needs far more than 49 videos and picks up lighting, background and camera differences instead of the face. Working from landmarks removes those differences. The TCN and the 3D-CNN tied in cross-validation (balanced accuracy 0.90 ± 0.06, pooled AUC 1.00, no false positives on the 22 unaffected videos), but on the new clips the TCN caught the most droops at less than half the 3D-CNN's size. So the TCN runs on every bed.
Live, the model's output is compared with the patient's own first 20 seconds and blended with the landmark score:
$$S = 1.25\,\frac{s_{\text{landmarks}}}{\tau} + \bigl(\ell_{\text{model}} - \tilde{\ell}_{\text{first 20 s}}\bigr), \qquad \text{alert if } S \ge 2 \text{ for 6 s}$$
Any 2-second window where the head turns more than 15° doesn't count toward the model's part of the score. On a benchmark of six clips with a synthetic one-sided droop, the blend caught 14/36 droops vs 6/36 for landmarks alone, with 0/6 false alerts.
Eyes. The gaze measure is how far the iris sits from the middle of the eye opening, divided by eye width, averaged over both eyes:
$$g = \frac{(\mathbf{c}_{\text{iris}} - \mathbf{m}_{\text{eye}}) \cdot \hat{\mathbf{u}}}{w_{\text{eye}}}$$
Eyes held more than $0.08$ off the patient's own centre for 30 s start the check automatically. In the check, a gaze palsy is flagged when the smaller sideways movement is under $0.45\times$ the larger one.
Arms. MediaPipe's Pose Landmarker gives wrist height above the shoulder, measured in shoulder widths. During a 10-second hold, an arm that drops by at least $0.20$ shoulder widths, and by $0.15$ more than the other arm, counts as drift.
Voice and system. Calm, pre-recorded ElevenLabs prompts guide the patient ("Without moving your head, look to your left"). A FastAPI server runs one worker per bed and streams state and video to the dashboard over WebSockets. iPhones join by scanning a QR code through a Cloudflare quick tunnel, with no app or account. We calibrated every threshold live on our own webcams.
Challenges we ran into
- Head turns look like droop. Turning the head makes the face look asymmetric on camera, to both the landmark detector and the learned model. We added pose correction, an optional guided head-turn calibration, gates that skip frames instead of guessing, and a 15° gate that pauses the model while the head is turned.
- Telling a gaze palsy from a head turn. Someone who turns their head instead of moving their eyes looks just like someone who can't move their eyes. We now track head rotation during the check and report "not assessed" rather than a false finding.
- False arm drift. Both arms getting tired, or the tracking jumping between frames, looked like one-sided weakness. Drift now has to be one-sided, the other arm must stay up, and jumpy tracking is rejected.
- Real-world networking. iPhone Safari only allows the camera over HTTPS, and campus Wi-Fi blocks phone-to-laptop traffic. A Cloudflare tunnel solved both.
- Small data. 49 training videos is tiny, so we used subject-level cross-validation and a synthetic-droop benchmark, and we report the uncertainty honestly. It's also why a raw-video transformer lost to models working from landmarks.
- Merging three parallel streams (the detector, the learned model, and the BE-FAST check and UI) in one weekend.
What we learned
- Compare with the patient, not a population. A personal baseline beats a population threshold: everyone's "normal" face is a little asymmetric.
- Saying "I couldn't measure that" is safer than guessing. Reporting "not assessed" or "inconclusive" beats a confident wrong answer.
- With little data, the right input beats a bigger model. Landmarks threw away lighting and background, so small models trained on them beat a transformer trained on raw video.
- Clinical workflow matters as much as the model. Acknowledgement, checklists, timers and audit logs are what make an alert useful to a nurse.
What's next
Testing on real ICU footage with clinicians, a baseline that adapts slowly over hours, speech (the "S" in BE-FAST), and integration with hospital paging and EHR systems.
Eye-CU is a research prototype, not a medical device. It supports clinical assessment and never replaces it.
Built With
- claude
- elevenlabs
- google-colab
- javascript
- python
- pytorch
- uvicorn