Vigilo

Inspiration

Cognitive fatigue is invisible until it's already cost you something — a bug you didn't catch, a typo in an email, a bad call in a meeting. Wearables track sleep and activity after the fact; nothing catches the fog while it's happening, using hardware everyone already has open: a laptop webcam.

What it does

Vigilo computes a live 0–100 Fog Score from webcam-only biometrics — blink rate, eye openness, head movement, brow tension — plus optional voice pacing (pause rate, pitch variance) if the mic is on. It calibrates to your resting baseline in 15 seconds, then flags live deviation from it, not a fixed threshold. Crossing into "foggy" triggers a targeted micro-intervention — guided breathing, an eye break, a stand-and-stretch prompt — and every session exports as CSV/JSON.

How we built it

React + TypeScript + Vite + Tailwind v4, zero backend. Face tracking via MediaPipe FaceLandmarker (on-device WASM, GPU delegate) — 478-point mesh, blendshapes, and a facial transformation matrix. From that: a 6-point EAR formula for blink/eye-openness, a threshold state machine for blink counting, rotation-matrix-to-Euler decomposition for head pose, and direction-reversal detection for nodding. Voice runs on the Web Audio API — a custom autocorrelation pitch tracker plus RMS-gated VAD. The scoring engine z-scores each live signal against the personal baseline, weights and sums them, and EMA-smooths the result — weights for inactive signals (e.g. mic off) are excluded dynamically so the score stays on the same scale either way.

Challenges we ran into

There's no universal "this blink rate means fatigue" — solved by calibrating per-person instead of hardcoding thresholds. Keeping ~15fps detection smooth in-browser with no server to offload to. Making the composite score comparable whether or not voice is active. Tuning EMA smoothing so the number reads calm, not jittery, without lagging behind real state changes.

Accomplishments that we're proud of

A fully real-time, zero-backend pipeline with sub-second latency and genuinely personal calibration. The voice signal degrades gracefully — the score is identical with or without it. And the "fog" isn't just a color: the whole UI gets faintly hazy as the live score climbs, tied directly to the data.

What we learned

Blendshapes are far more robust than raw landmark math for expression signals. Browser-only pitch detection is workable but noisy without a heavier algorithm (YIN, CREPE). Dynamic-weight scoring is a clean way to handle optional inputs without special-casing the UI for every combination.

what's next for Vigilo

Fit the score weights against labeled fatigue data instead of hand-tuning them; a lightweight backend for cross-device history and trend reports; a subtle OS-level nudge (screen dim, notification) when fog sustains; code-splitting the MediaPipe bundle for faster load; a small user study validating the score against self-reported fatigue.

Built With

• gpu-delegate
• mediapipe-landfacemarker
• react
• tailwind
• vite
• wasm
• web-speech-audio