Main Menu
Fighting the enemy bot
weapons

Inspiration

We wanted to build a fighting game you play with your whole body, not just a controller. Inspired by arcade mecha games and motion-controlled VR, we asked: what if you could pilot a giant robot using your actual movements, a dance pad, and a physical sword?

What it does

A first-person mecha fighting game where your webcam tracks your head to aim, an Arduino dance pad moves you around the arena, and hand to swing your sword. Voice commands let you cast spells and turn around. Multiplayer lets two players fight in real time.

How we built it

Three.js and React Three Fiber for 3D rendering, MediaPipe PoseLandmarker for body/head tracking via webcam, an Arduino wired to foot buttons sending serial data mapped to WASD via a Python bridge, WebSocket server for multiplayer sync, and Zustand for state management.

Challenges we ran into

Getting CV tracking to feel responsive without jitter. Color detection for different props (sword, gun, fire button) overlapping in hue space. Syncing the Arduino serial input with the browser game. Making head tracking feel natural when your head has limited rotation range compared to a mouse.

Accomplishments that we're proud of

Head tracking runs in the render loop with zero-latency ref-based updates (no React re-renders). The dance pad works seamlessly as WASD input. You can genuinely sword fight another person using physical movements.

What we learned

How to run CV detection synchronously inside a game loop, how MediaPipe pose landmarks map to useful game inputs, how to bridge Arduino hardware to a web app, and that HSV color thresholding is surprisingly effective for real-time prop tracking.

What's next for SUPERMECHAFIGHTER ULTRA REALITY 3600 OF DOOM

IR camera tracking for more precise aiming, more weapons and spells, haptic feedback through the physical props, tournament/spectator mode, and additional arenas with interactive environments.