Let Him Rook

Inspiration

What if a chess piece could fight back? We wanted to combine chess strategy with the energy of an arena game, using a webcam to turn players’ movements into the controls.

What it does

Let Him Rook is a multiplayer 3D chess game where every attempted capture starts a real-time battle.

Players pinch to move pieces, use body movement to navigate the arena, and perform hand gestures to attack. Each piece has a distinct fighting style: sword-and-shield pawns, rifle-wielding rooks, mounted knights, ice witches, fire wizards, and kings that summon blue clones.

The battle determines whether the capture succeeds. Players can also practice against a bot, request AI advice, and review fight notes showing what happened and how to improve.

How we built it

We built the game with TypeScript, React, Three.js, and React Three Fiber. MediaPipe tracks hands and body landmarks locally, keeping webcam video on the player’s device.

A Node.js WebSocket server validates chess moves, runs combat, and synchronizes both players. Muse Spark powers optional AI advice, with built-in advice available if the API fails.

We host the game on Vultr at pawnhub.tech.

Challenges we ran into

Reliable gestures were our biggest challenge. Pinches could release unexpectedly, movement varied between players, and tracking competed with 3D rendering for limited graphics power.

We added personal calibration, smoother tracking, gesture confirmation, and performance improvements. We also had to make chess rules and real-time combat work together while keeping both players synchronized.

Accomplishments that we're proud of

We brought a complete loop together: move a chess piece, enter a battle, fight through webcam gestures, and return the result to the board.

We also gave every piece its own character and ability, built online multiplayer, and added practice tools and fight reviews to help players learn.

What we learned

Gesture controls need clear feedback and room for human error. We learned how to balance tracking accuracy, responsiveness, and rendering performance, and how to keep multiplayer outcomes consistent through server-controlled game logic.

What's next for Let Him Rook

We want to improve gesture reliability across more devices, refine character animations, balance the six classes, and expand the AI coaching using fight history.