Helm

Inspiration

Every adaptive controller on the market still assumes you can press something. Switch hubs need a body part that can push a button. Sip-and-puff and mouth joysticks need a lip seal and enough breath. Face-tracking tools exist, but they act as a mouse and keyboard on fixed thresholds, which is built for pointing, not for steering a kart. Eye tracking asks for the one thing a player needs free: their eyes.

Many people with limited hand and arm movement, from spinal cord injury, ALS, cerebral palsy, muscular dystrophy and other conditions, can still turn their head a little and move parts of their face, sometimes on one side only. Games are one of the places they're most often shut out. We built Helm for them: a controller that starts from whatever movement a person has, instead of asking them to adapt to it.

What it does

Helm turns a regular webcam into a full Xbox 360 controller, built for players who can't use their hands.

Head movement is the analog stick. A thirty-second calibration measures how far you can comfortably turn, so a small movement can still mean full stick. Someone who can only turn a few degrees gets the same full range as anyone else.
Expressions are buttons and triggers. An eyebrow raise presses A. A half-open jaw is half throttle. A nod, a smile or a pucker can each be a button.
It learns your gestures. Show Helm any movement five times, then talk and look around so it learns what not to fire on. A one-sided smile works as well as any other, which matters for players with movement on one side of the face only.
Breath is one more input. A straw on a pressure sensor adds soft and hard sips and puffs, the same inputs many sip-and-puff users already know.
A coach builds your mapping. An AI agent reads the game's controls and the gestures that were reliable for you, drafts a mapping, checks it against the rules, and revises it after a session.
It keeps adapting. Fatigue is real. If a gesture weakens as you tire, Helm lowers its threshold and tells you, instead of silently failing.
It remembers you. Each player gets a saved profile with their calibration and their moves, and a profile only responds to the face that enrolled it, so a carer or friend leaning into frame doesn't take over the controls.

Helm appears to Windows as a real controller, so it works with any game that accepts one, with no mods. Closing your eyes for two seconds pauses it. Everything runs on the laptop: video stays local and is saved only when dev mode is explicitly enabled for debugging.

The dashboard is built to be usable by the people it's for: calibration runs with no mouse and no keyboard, the text is large and high-contrast with a text-size control, every input lights up as it fires, and a reduced-motion setting turns off all animation.

How we built it

One Python process and a React dashboard.

Vision: OpenCV capture into MediaPipe Face Landmarker, which gives head pose and 52 expression scores per frame.
Signal: each player's neutral pose and range are removed and scaled, then smoothed with a One Euro filter.
Gestures: thresholds with hysteresis, minimum hold and a refractory period, plus a personalized logistic classifier (scikit-learn) with feature selection, a "none" class, distance rejection and temporal confirmation for learned gestures.
Output: a virtual Xbox 360 controller through vgamepad and ViGEmBus, with keyboard emulation as a fallback.
Breath: an ESP32 streams raw pressure over USB serial; all thresholds live on the laptop, per player.
Coach: a LangChain deepagents agent with four tools (read the game's controls, read the player report, check a mapping, submit a mapping). It has an offline default so the demo never depends on Wi-Fi.
Dashboard: FastAPI WebSocket to React, Vite and TypeScript. It runs calibration with no clicks and shows every input as it fires.

Control stages hand off through "latest value" slots, so a slow consumer drops stale frames instead of adding lag. Optional dev recording uses a bounded background queue. Every frame is timestamped at capture, after inference and after the controller write, so latency is measured rather than estimated.

Challenges we ran into

Smoothing versus lag. The filter settings we started with lagged a head turn by about a quarter of a second in simulation. Steering needed a much more responsive setting, with the dead zone absorbing the jitter instead.
Left and right. The expression scores refer to the player's own face, so mirroring the camera image before tracking silently swaps them. Helm mirrors only the preview.
Breath events are instantaneous. A puff arrives as a single event, so the minimum-hold rule that protects expression buttons would have blocked every breath command. Breath needed its own rule.
Talking. Speech moves the same muscles as deliberate gestures. That is why every player records a "don't fire" class.
A dashboard you can use without hands. Most UI assumes a mouse. We went through many dashboard designs to get one where calibration needs no clicks, the most important information (which move presses which button) sits at the top, and everything stays readable from a distance. Early versions were too small and too low-contrast to read from a chair.
Calibration that doesn't overwhelm. A full dashboard is too much to take in while you're being asked to hold still. During calibration Helm now blurs everything you don't need yet and shows one instruction at a time in the middle of the screen.

Accomplishments that we're proud of

Calibration runs with no mouse and no keyboard, start to finish.
A dashboard designed for accessibility from the start: large high-contrast text, a text-size control, a reduced-motion mode, and a calibration screen that focuses you on one step at a time.
Per-player profiles locked to the enrolled face, so the controller only listens to the person who's playing.

What we learned

Measuring is a feature. Stamping every frame made tuning a matter of reading numbers.
Per-player scaling matters more than a better model: the same tracker feels unusable or natural depending on whether its range matches the person.
Accessibility has to cover the whole experience, not just the input. A face-controlled game is no use if setting it up needs a mouse, so every step, from picking a profile to calibrating to pausing, needs a path that works without hands.
Less on screen is more. Each time we removed something from the dashboard, it got easier to understand at a glance.

What's next for HELM

Testing with disabled players. Helm is built for people with limited hand and arm movement, but no one who tried it this weekend has a motor impairment; our restricted-range test is a proxy, not a substitute. The first step is working with organizations such as AbleGamers and Makers Making Change to put it in front of the players it's for, and tuning it with their feedback.
An open-source release, a Linux backend, a mouse-and-keyboard mode, and a shared library of per-game profiles.
Beyond games: the same pipeline for desktop navigation and communication shortcuts.
Helm stays free for players.