Red Dead Dimension

A Wild West train shooter where your hand is the gun. Built at SteelHacks.

You're on the roof of a moving steam train and bandits keep coming. Make a finger gun at your webcam and point. That's your crosshair. There's no controller.

You do	Game does
Point a finger gun	Aim
Drop your thumb	Fire
Slap the gun hand with your other hand	Reload
Lean / duck for real	Dodge bullets and tunnels
Lower your arm	Holster

You have three hats. Each lap gets faster. Lose all three and it's over.

How it works

webcam -> MediaPipe hand + body landmarks -> tracker (Python) -> OSC -> Unreal Engine 5
                                                 ^
                                       Arduino glove (optional)

We didn't train a model. MediaPipe gives us hand and body skeletons, and everything after that is our own geometry:

Real-world scale from one RGB camera. Palm bones don't stretch, so how big they look tells us how far away the hand is. Aim is measured in actual centimetres of fingertip travel, so it feels the same sitting at a laptop or standing across a room.
Aim is fingertip position, not finger direction. We tried casting a ray along the finger. A finger pointed at a camera is basically a dot, so the direction is useless. Position is steady to about 2 mm.
Shots rewind. Pulling the trigger jerks your hand. We keep a second of aim history and fire where you were pointing just before the jerk.
One Euro filter on the aim: heavy smoothing when you hold still, almost none when you move fast.
Crowd-proof. A hand only counts if it's attached to the player's arm and isn't farther away than their chest. Spectators behind you can't steal the aim.

Every threshold in tracker/config.py marked [rec] came from replaying recorded sessions, not guessing. The pipeline has no camera or network code in it, so recordings replay through it exactly.

More detail in docs/ARCHITECTURE.md.

The glove

Optional. An Arduino with a switch under the thumb, a buzzer and an LED. The click gives an exact trigger time and you feel a buzz when you get hit. Firmware is in arduino/finger_gun_glove/. The game plays fine without it.

Running it

You need Unreal Engine 5.8, Python 3.10-3.12 and a webcam.

cd tracker
python -m venv .venv
.venv/bin/pip install -r requirements.txt

The two MediaPipe model files go in tracker/models/. The tracker prints the download links if they're missing.

Open unreal/FingerGunGame/FingerGunGame.uproject once and let it build. Then:

./play.sh

On Windows:

powershell -ExecutionPolicy Bypass -File .\play.ps1

This starts the game and the tracker together. The game also works with mouse and keyboard if the tracker isn't running.

Useful tracker flags (run from tracker/):

python run.py                          # tuning window with the skeleton overlay
python run.py --record                 # save landmarks to a .jsonl
python run.py --replay recordings/x.jsonl
python run.py --set aim_span_m=0.5     # override any config value

Don't upgrade mediapipe past 0.10.21. Newer versions crash or leak memory on macOS.

Layout

tracker/   Python computer vision + gesture logic
unreal/    UE5 game
arduino/   glove firmware
art/       Blender sources and exported models
docs/      architecture notes and slides