Gesture Kombat

Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

What we learned

What's next for Gesture Kombat

Inspiration

What if your body could become the controller for a fighting game? Inspired by the energy of arcade combat games, Gesture Kombat explores a more physical way to play: use an ordinary webcam to turn real movements into character actions.

What it does

Gesture Kombat is a browser-based 3D fighting game controlled through body movements and hand gestures. Players can punch, kick, block, crouch, jump, move sideways, and activate special powers.

Compatible actions work together, allowing combinations such as moving while punching. The game includes AI opponents, random online matchmaking, private friend duels, and guided training.

A dedicated Live Tracking mode displays the player's camera beside a selected 3D character that follows their tracked movements. Players can choose between Raven and Vortex and three arena backgrounds. Keyboard and touch controls provide alternative inputs.

How we built it

The web application uses TypeScript, React, Next.js-compatible Vinext tooling, and Three.js for 3D rendering. MediaPipe Holistic provides pretrained body and hand landmark tracking. Custom gesture logic uses motion history, normalized body geometry, confidence thresholds, smoothing, and independent cooldowns.

Camera processing happens on the player's device. Recognized actions drive gameplay, while tracked joint directions animate the character skeleton. Multiplayer APIs and Cloudflare D1 manage matchmaking, private rooms, health, energy, damage, and match outcomes.

The initial computer vision prototype used Python, OpenCV, and MediaPipe. The browser version integrates tracking directly into the game.

ChatGPT/Codex assisted with implementation, debugging, and documentation. Character models were created using Tripo Studio and adapted for the game. We integrated pretrained tracking models rather than training a custom machine learning model.

Challenges we ran into

The biggest challenge was distinguishing intentional gestures from ordinary movement. Static poses alone were insufficient, so recognition needed to consider movement over time.

Supporting simultaneous actions required independent detection and cooldowns. Balancing camera inference with 3D rendering also required performance adjustments.

The imported character models needed optimization and skin-weight repairs before their limbs could respond to tracking. Automatic skinning made animation possible, but cape, armor, and finger deformation still need refinement.

Accomplishments that we're proud of

We connected computer vision, animated 3D characters, and multiplayer game logic in one publicly deployed application.

The prototype supports simultaneous gestures, guided training, split-screen live tracking, AI matches, and online play. The current version passed 40 automated regression checks and a production build.

What we learned

We learned how pose detection, gesture recognition, and character animation solve different parts of the same interaction. Reliable movement controls require careful handling of timing, visibility, body proportions, and tracking noise.

We also gained experience coordinating browser camera access, asynchronous model loading, 3D rendering, and server-controlled multiplayer state.

What's next for Gesture Kombat

We plan to refine character skinning and combat animations, improve calibration and recognition reliability, and measure tracking performance across different devices.

Further work includes better cape and hand movement, improved multiplayer synchronization, and additional usability testing. Our goal is a more responsive and polished experience while keeping webcam processing local.

Built With

• cloudflare-d1
• css
• html
• javascript
• mediapipe
• next.js
• opencv
• python
• react
• three.js
• tripo-studio
• typescript
• vinext
• webgl