One sign sets up the meeting: invite, email, time, done.
Custom gestures so you can work "fast as hell"
Real time captions

FastASL

Run your Mac in American Sign Language.

FastASL is an always-on overlay that reads signs from your webcam, turns them into real actions (email, calendar, FaceTime, apps), and burns live English captions into your FaceTime video — so a Deaf user can drive their Mac entirely in ASL.

Built for Cal Hacks 13.0 · Electron + MediaPipe + Factory Droid + Ollama

Inspiration

A Deaf user shouldn't need a keyboard, a voice, or a hearing intermediary to run their own computer. FastASL started from a simple question: what if signing to your Mac just... worked? — always available, on top of everything, reading ASL and acting on it directly.

We also wanted captions that travel with the person, not just on their own screen: what you sign should show up as English inside the FaceTime video your caller receives.

What it does

Always-on overlay. A small card floats over every app with a live camera view and a chat. One click hides it; an open palm held up brings it back.
Signs stream into sentences. Words appear the moment each sign lands (TOMORROW, MEETING, WHAT, TIME...), building a sentence you send by lowering your hands.
Signs become actions. An agent (Factory Droid) reads the signed sentence and uses real Mac tools: Calendar, Mail, Messages, Contacts, Notes, Spotify, FaceTime, apps, web search, screenshots.
Safe by default. Anything that sends, creates or calls shows a Confirm card first — a thumbs up runs it, a fist cancels.
MEETING FRIEND. Two signs, then hold up 1–4 fingers for a time: Mail and Calendar open side by side, the invite and the email go out, and both apps show what was sent.
CALL FRIEND. One confirmation and FaceTime calls the contact, pressing Call by itself.
Live captions on FaceTime. What you sign is captioned into the video your caller receives, rewritten from ASL word order into English (TOMORROW MEETING WHAT TIME → "What time is the meeting tomorrow?").
Custom gestures. Record any hand shape and bind it to a task — e.g. one gesture opens six Claude Code terminals tiled across the screen.
Contact nicknames by sign or initials. FRIEND, or fingerspelled initials like D W, stand for a saved contact.
Gesture controls. Thumbs up for a quick menu, swipes to switch modes, finger counts to pick options, swipe up for Settings.

How we built it

Signs are read on the Mac; only the agent's task goes to the cloud.

Layer	What we used
App shell	Electron 39, electron-vite, React 19, TypeScript; transparent always-on-top overlay window
Hand, face & body tracking	Google MediaPipe Tasks Vision (Hand, Face, Pose Landmarkers), GPU, alternating frames to hold ~30 fps
Streaming sign recognition	Our own rules over the landmarks (handshape, location relative to the face, movement): a dictionary of signs plus per-word rules, firing mid-movement so words stream in
Pretrained sign models	1st-place model from Google's Kaggle Isolated Sign Language Recognition competition, ported to ONNX (11 MB, fp16), run with onnxruntime-web in a worker; Microsoft's ASL Citizen pose model (ST-GCN) for a larger vocabulary
Control gestures	Our gesture engine: open palm, fist, thumbs up, swipes, finger counts, plus custom hand shapes
Agent	Factory Droid SDK (`@factory/droid-sdk`, the droid CLI), one persistent session; Droid's own tools disabled, ours exposed via an MCP server
Mac actions	19 tools in AppleScript / JXA: Calendar, Contacts, Mail, Messages, Notes, Spotify, media keys, FaceTime, open app, web search, type text, screenshot, Claude Code terminals, meeting invite
Fast paths	CALL / MEETING + nickname and custom gestures run fixed tool calls with no model — about a second
Live captions	1280×720 caption window (camera + captions) → OBS Window Capture → OBS Virtual Camera → FaceTime
ASL → English	Local Ollama llama3.2 (3B) rewrites signs into one sentence in 0.2–0.9 s; used only if every signed word survives

We built it in phases, each tested before the next — and rebuilt recognition twice when live signing showed where it broke:

Overlay and controls — floating card, MediaPipe pipeline, gesture engine, Settings, and a test suite that grew past 1,300 tests.
First recognition: teach your own signs — users recorded each sign; an LLM turned words into intent. Live, teaching was slow and unreliable.
Pretrained sign models — switched to models trained on thousands of signers (Kaggle ISLR winner ported to ONNX, Microsoft ASL Citizen), so no teaching is needed.
Real actions — Factory Droid SDK with our own Mac tools, a conversational chat, and Confirm cards for anything that sends.
Streaming recognition — rules that recognize demo words mid-movement and build the sentence live, checked against ASL Citizen clips.
Live captions for FaceTime — caption camera window routed through OBS virtual camera, plus a local model that fixes ASL word order.
Fast paths for the demo — CALL FRIEND / MEETING FRIEND skip the model; initials name a contact; a custom gesture opens six Claude Code terminals.

Challenges

Most of our time went into making recognition hold up on a real person in a busy hackathon hall.

Problem	What we did
Teaching signs per user was slow and unreliable	Moved to pretrained models (Kaggle ISLR, ASL Citizen) plus hand-written streaming rules
The body tracker locked onto people behind the signer; face and pose dropped out between frames	Track several people, keep the closest; hold the last face and pose across gaps
Recognition waited for a whole movement	Streaming rules fire mid-sign, so the sentence builds as you sign
FRIEND's two hooks also read as TIME ("time friend time friend")	A FRIEND right after a TIME replaces it; a TIME right after FRIEND is dropped; fingerspelled initials as fallback
A resting fist hid the card and stopped recognition (and the captions)	The fist no longer hides the card while signing words; the card has a button instead
"Listening" flickered off and on between tries	It stays up 2.5 s for the next try
Loosening MEETING's handshape check fires on WHAT and EMAIL	Measured both on recorded clips; kept the stricter check
Gemini's quota ran out mid-hackathon, then its model was withdrawn	Factory Droid reads the signs itself; English captions run on a local model
Droid wouldn't call our tools	It calls MCP tools through its Script tool, so our permission check now reads the calls inside a script
FaceTime stops at "Click to call"	After the user confirms, fastASL presses Call through macOS accessibility
FaceTime has no caption API	Captions drawn into a window that OBS publishes as a virtual camera
Once OBS's virtual camera existed, fastASL picked it as its own camera and filmed itself (a black loop)	Default to the built-in camera, never a virtual one
A hidden second copy of the overlay in the caption window declined every Confirm	The caption window loads only its own code
At 3% battery macOS throttled the GPU; tracking fell to 3–9 fps	Plug in for demos; it recovered to ~30 fps
A small model guessed (YES became "I'm okay")	An English caption is used only if every signed word is in it

Accomplishments

Everything below was measured — on recorded ASL clips, or live on our laptop.

Measure	Result
FRIEND recognized, ASL Citizen signers (videos run through our tracker)	25 of 29, ~0.6 s after the hands appear
FRIEND firing on other signs	1 of 895 clips
TIME firing on FRIEND clips (before → after fix)	8 of 58 → 2 of 58
MEETING recognized on recorded clips	9 of 12 and 26 of 29 (two clip sets)
MEETING firing on other signs	0.2% and 1.4%
ASL → English captions on 16 demo phrases	15 correct, 0.2–0.9 s each; the 16th fell back to the signs
Hand tracking on an M-series MacBook	~6 ms/frame at ~30 fps
CALL FRIEND / MEETING FRIEND before the Confirm card	~1 s, no model involved
Automated tests	1,325 passing

What we learned

Recorded datasets flatter a recognizer. A live signer in a busy room, a hand resting in a fist, a dying battery — that's what breaks it. We tested every fix on the real laptop.
ASL is not English in signs. Word order differs and one handshape starts many signs, so sequence rules (FRIEND over TIME) mattered as much as per-sign rules.
For a Deaf user, speed and trust beat cleverness. Fixed fast paths for common commands, and one clear Confirm before anything reaches another person.

What's next

More streaming signs, trained on more signers, and the full fingerspelling alphabet without teaching.
Captions in Zoom, Meet and Teams the same way, plus a voice-to-text return channel for the hearing caller.
A signed app instead of a dev build, and a native camera extension instead of OBS.