Motif

Inspiration

For people living with ALS, the way they control technology keeps changing as the disease progresses. A movement that works today may not work in six months, and every change means another assessment and another setup. Most assistive tools ask people to move the way the tool expects. We asked the opposite question: what if the tool learned how you can move, and was quick to re-learn when that changes?

What it does

Motif uses an ordinary webcam to learn one to four facial movements that a person can make comfortably and reliably, such as an eyebrow raise, opening the mouth or a long blink. It then turns those movements into controls for a communication board:

One movement: the board scans automatically, and the movement selects.
Two movements: one steps to the next item and the other selects, at the person's own pace.
Three or four movements: going back and pausing get their own movements too.

The person builds a message from large phrase tiles, then selects Play to speak it aloud. Undo, Say it again, saved Favourites and Pause are always within reach. A paired caregiver can receive requests like "I need water" on their own device and confirm they've seen them.

Setup takes about a minute and a half:

a 10-second rest check;
choose a movement, or let Motif suggest one;
three recorded attempts;
two separate checks;
10 seconds of normal activity, such as talking and looking around, where nothing may trigger.

Nothing turns on until the person or their helper approves it. If a movement becomes tiring, they can swap it from the Helper menu without starting over.

How we built it

Motif runs entirely in the browser. MediaPipe Face Landmarker tracks the face locally, and video never leaves the device. Our own TypeScript engine turns the landmark data into movement signals and learns each person's thresholds from their resting state and their recorded attempts.

For each movement, let $r$ be the person's resting ceiling (the 95th-percentile resting variation), $w$ their weakest recorded attempt and $\sigma$ their resting noise. The margin between rest and movement is

$$ m = w - r $$

A movement is accepted only if it stands clearly above rest:

$$ m \ge 3\sigma $$

It then fires above an activation threshold and must drop below a lower release threshold before it can fire again:

$$ \theta_{\text{on}} = r + 0.65\,m, \qquad \theta_{\text{off}} = r + 0.25\,m $$

Separate on and off thresholds stop a shaky hold from double-triggering. The signal must also stay above $\theta_{\text{on}}$ for at least 250 ms (500 ms for winks), so brief twitches are ignored. Each hold performs one action, and lost or uncertain tracking performs none.

The interface is React and TypeScript, built with Vite and Tailwind. Speech uses the browser's on-device voices. Caregiver pairing and requests go through Supabase, and that is the only data that leaves the device. Movement settings stay in local storage. We tested with Vitest and Playwright, using recorded and synthetic movement data instead of a live camera.

Challenges we ran into

Talking looks like gesturing. Mouth and eyebrow movements happen all the time in normal speech. We added the everyday-activity test, which a movement must pass before it can be approved, and we require a return to rest before the next action.
Telling similar movements apart. A blink and a one-eye wink overlap. When someone adds a second movement, Motif checks that each one doesn't set off the other.
Too many screens. Our first version had a separate page for every step. People found it confusing, so we merged setup into one guided flow and moved settings behind a single Helper button.
Recovering from mistakes. One accidental trigger used to wipe the whole message. Now Undo removes only the last phrase, and Clear asks before it acts.

Accomplishments that we're proud of

A person can go from opening the page to speaking a sentence in about two minutes, using only one facial movement.
Camera processing stays entirely on the device.
The same calibrated movements already drive a prototype that scrolls web pages.

What we learned

Accessibility design is mostly about reducing effort and recovering from errors, not adding features.
A movement that works every time matters more than an impressive one.
Every extra screen or click costs more for someone with one slow switch than it does for us.

A note on testing

Our demo uses teammates simulating different ability levels. Before real-world use, Motif would need to be validated with people living with ALS and the clinicians who support them.

What's next

Browsing the web with the same movements. We prototyped a Chrome side-panel extension that runs Motif next to a webpage. With the same calibrated movements, a person can scroll up and down, go back, and return to their messages, on sites a helper has allowed. Automated tests in a test build of Chrome show the loop working with a stand-in for the gesture input. The next step is connecting it to the full app and testing it with real users.
Watching for changes over time: tracking how reliable each movement stays during use, and suggesting a re-check when one starts to fade.
Testing with occupational therapists and speech-language pathologists.
A letter board with word prediction for messages beyond the phrase list.
Personal voices through voice banking.