Inspiration
Almost every game assumes you have two working hands, ten fingers and a controller. If an injury, a disability or limited fine motor control rules that out, most games are closed to you, and adaptive hardware is expensive and hard to find. Every laptop already has a webcam, so we asked: what if your hands were the controller, and the controls adapted to you instead of the other way around?
What it does
OpenHand lets you play PC games with your bare hands in front of a webcam, with no controller or special hardware.
- Invisible steering wheel: grip an imaginary wheel with both hands and turn it to drive PolyTrack. Steering is proportional, so a slight turn of your hands gives a slight turn in the game. Holding your hands up accelerates and opening both palms brakes.
- Hand mouse: move the macOS pointer with one hand, and pinch to click and drag in any app.
- Controls built around the player: type what you can do, like "left hand only, can't make a fist", and Gemini builds a control scheme from gestures you can actually perform. Feedback like "steering is too twitchy" tunes it further.
A live dashboard shows the camera feed with hand landmarks, a steering wheel that turns with your hands, the active gestures and the current mapping.
How we built it
The pipeline goes from webcam, to MediaPipe hand tracking, to gesture signals, to a JSON mapping, to the operating system's keyboard and mouse.
- Tracking: OpenCV captures frames, and MediaPipe's Hand Landmarker finds up to two hands with 21 points each.
- Gestures: Python functions turn landmarks into signals such as steer, tilt, pinch, fist and pointer position. Every distance is divided by hand size, so gestures work at any distance from the camera.
- Filtering: a One Euro filter smooths analog signals, staying steady when you move slowly and responsive when you move fast. Hysteresis on the on/off gestures stops flicker.
- Mappings: JSON files bind each gesture to a control with a dead zone, gain and response curve. Files reload automatically when saved.
- Analog steering on a keyboard game: PolyTrack only has arrow keys, so we pulse them. If you turn the wheel 30% of the way, the key is held down 30% of the time. A slight turn sends sparse taps, a bigger turn sends faster taps and then longer holds, and full lock holds the key down. No tap or gap is ever shorter than one game frame, so no press is missed.
- Gemini: structured output is constrained to our Pydantic schema, so the model can only pick gestures and keys that actually exist. Our code then rejects any binding that uses a hand the player said they can't use, and sends the errors back to Gemini for a retry. Good results are cached so the demo also works offline.
- App: a FastAPI server streams the preview and live state, switches profiles and runs calibration. The front end is plain HTML and JavaScript, and keyboard and mouse output goes through pynput.
Challenges we ran into
- Pivoting mid-hackathon: we started with Mario Kart Wii on the Dolphin emulator, but the setup was too fragile to demo reliably. On Saturday evening we switched to PolyTrack, which meant getting analog steering out of plain arrow keys.
- Fists that clicked: in a fist the thumb rests against the curled index finger, which looks exactly like a pinch. We now only count a pinch when the index finger is extended to meet the thumb.
- Jitter vs. lag: raw landmarks shake, but heavy smoothing makes steering feel slow. Tuning the filter and dead zones took many test laps.
- Trusting an LLM with accessibility: a model that is usually right isn't good enough when a wrong answer means a control the player physically can't use. That is why the hand rules are enforced in code, not just in the prompt.
Accomplishments that we're proud of
- Driving full laps of a real game with an invisible wheel, with proportional steering that feels right.
- Typing "left hand only" and getting a playable one-handed scheme back in seconds.
- Safety by default: control starts paused, and if your hands leave the frame every key is released within 150 ms.
- 20 unit tests that run on synthetic hands, plus record and replay of real sessions so we could tune without a camera.
What we learned
- Reliable gestures come mostly from normalization, filtering and hysteresis, not fancy models.
- Pulsing a key, like PWM in electronics, can make a digital input behave like an analog one.
- LLMs work best as configuration generators inside strict guardrails: schema-constrained output, hard checks in code, and a retry that feeds the errors back.
What's next for OpenHand
- Profiles for more games, including ones that use mouse and keyboard together.
- Support for Windows and Linux.
- Head and face gestures for players with limited hand mobility.
- Testing with players with motor disabilities, with occupational therapists helping shape the default schemes.
- A shareable library of community-made control schemes.
