ComboCraft

Inspiration

ComboCraft started on a whim during a late-night gaming session. We were playing a game similar to Wii Boxing when someone asked a question that changed the direction of our whole weekend: what if we could be the characters on the screen fighting in real time? We began experimenting with basic punch detection, expecting to build a simple two-player boxing prototype. As soon as we saw how well the gestures registered, the idea shifted. The webcam could recognize more nuanced movements than we expected, and those motions felt expressive enough to support actual abilities rather than single hits. Once that clicked, a boxing clone no longer felt interesting. Since our team enjoys game development, the project naturally grew into something more ambitious. We started designing elemental spells, resource systems, and a multi-stage boss encounter that reacts to the player’s movements. A small “what if” moment turned into a chance to see how far we could push real-time computer vision inside a browser game.

What it does

ComboCraft is a gesture-based browser game where real movements cast elemental spells in real time. A fist triggers a fireball, an open hand forms an ice shard, a pointed gesture creates lightning, and combinations of these gestures create multi-step combo attacks. The name reflects this core mechanic: players craft stronger abilities by chaining gestures together.

The game takes place in the Inferno Realm, where the player faces a Demon Boss that evolves as its health drops. Each phase brings new animations, attacks, and behavior changes. The game includes three difficulty settings, allowing players to choose between easy, medium, and hard encounters that scale the boss’s aggression and attack patterns. Mana and healing orbs appear as fragments of the boss’s energy, and players break them for resources. Surviving the fight requires timing gestures, managing resources, positioning, and adapting to the boss’s increasingly complex phases.

The interaction system behind ComboCraft creates room for expansion. The same spell casting and gesture could grow into a fitness-oriented experience that encourages movement through combat challenges. As the system develops, it could lead to fully physical encounters that involve dodging, ducking, kicking, and reacting to the boss with whole body movement.

How we built it

Gesture detection is powered by MediaPipe Hands, paired with MediaPipe Selfie Segmentation for background processing. To improve consistency and reduce gesture confusion, we trained a small sklearn model on custom samples. The backend is written in Python using Flask and OpenCV. It processes webcam frames, classifies gestures, and sends simplified commands to the frontend. The game itself runs entirely in JavaScript, HTML, and CSS. We used the canvas for boss animations, spell rendering, hit reactions, orb interactions, and UI effects. All health bars, mana bars, and pixel art integrations were built manually. Cursor AI was utilized to streamline repetitive animation formatting while the core systems were coded by hand.

Challenges we ran into

Running two MediaPipe models at once created a noticeable performance burden. Background removal required extra canvas operations every frame, which made timing more sensitive. Our first approach polled the backend at 60 frames per second, which resulted in stuttering and unpredictable gesture timing. Early gestures also overlapped visually and caused misclassification. We had to redesign the gesture set and rebuild the classifier to reduce ambiguity. Building the multi-phase boss required careful synchronization between gesture timing, animation states, and hit detection. Integrating all of these systems into a single, reliable game loop took multiple iterations.

Accomplishments that we're proud of

We are proud that ComboCraft feels like a complete and coherent game. The gesture system is responsive after tuning, the boss transitions create real progression, and the orb system adds strategy to the encounter. The UI and tutorial help the experience feel polished rather than experimental. This was the first time many of us used computer vision in a game setting. Bringing together CV processing, custom animations, game logic, resource systems, and phase transitions within a short timeline was a significant achievement for our team.

What we learned

We learned how to work with MediaPipe at a deeper level and how to improve gesture recognition through both model training and system design. We gained experience building timing based animation systems without a game engine, managing performance constraints, and coordinating multiple subsystems that rely on consistent frame updates. We also learned how to design gameplay around the limitations and strengths of real time computer vision, which required a different mindset than traditional control schemes.

What's next for ComboCraft

We plan to build a deeper combat system by expanding the combo mechanic. New elemental sequences will allow players to chain gestures into advanced abilities, making timing and decision making more important as fights become more demanding. Future content includes additional bosses with unique phase structures, minion encounters, and a progression system for unlocking new spells and combo paths. Difficulty will also become more dynamic, including scaling patterns that adapt to the player’s performance. On the technical side, we aim to replace backend polling with WebSockets to improve responsiveness. Our long term goal is full body computer vision so players can dodge, duck, kick, and physically move through the fight. ComboCraft started with hand gestures, but we see it growing into a full body, motion driven combat experience.

Built With

• css
• flask
• html
• javascript
• mediapipehands
• mediapipeselfiesegmentation
• opencv
• python
• scikit-learn