Inspiration

Our project, JutsuVerse, was inspired by the theme “Blocks That Connect the World.”

When we thought about what connects people around the world, we realized that these “blocks” do not have to be physical or technological. Culture can also be a block that brings people together.

Anime culture plays an important role in modern entertainment and has created communities across countries, languages, and backgrounds. Two people may come from completely different parts of the world and may not even speak the same language, but they might have grown up watching the same anime, recognizing the same characters, and remembering the same iconic moments.

For our team, one of those shared cultural experiences is Naruto.

That inspired us to create JutsuVerse, a real-time two-player web game where players use their webcams and perform hand gestures to battle each other. Instead of simply pressing buttons, players physically perform hand signs to execute different ninjutsu.

Our goal is not only to create a game, but to create an experience that can help strangers interact through something they already share. Two players who have never met before can enter the same room, turn on their cameras, recognize something familiar from anime culture, and immediately start playing together.

We also expanded this idea beyond anime. We added additional hand gestures inspired by memes and internet culture from different communities. Memes, like anime, often cross geographical and language barriers. By combining these cultural references into one game, we wanted JutsuVerse to represent how small pieces of shared culture can become blocks that connect people around the world.

What It Does

JutsuVerse is a real-time, two-player hand-gesture battle game.

One player creates a room and another player joins using the room code. Once both players are connected, they enable their cameras and battle each other by performing recognized hand signs.

The game recognizes the player's gestures through their webcam and maps valid gestures to in-game actions. Players can perform different types of ninjutsu, defend themselves, and react to their opponent in real time.

Different elemental ninjutsu interact with each other through a counter system, making the game about more than simply performing gestures quickly. Players also need to decide which ninjutsu to use and when to attack or defend.

The result is a game that combines computer vision, real-time networking, physical interaction, strategy, and shared culture.

How We Built It

JutsuVerse consists of a browser-based frontend and an authoritative game server.

For hand-sign recognition, we use a YOLOX-based ONNX model running directly in the browser. Webcam frames are processed locally, and recognized hand signs are converted into game actions. Running inference on the client reduces the need to continuously upload camera frames to the server.

For real-time multiplayer communication, we use WebSockets. The backend is built with FastAPI and manages player rooms, game state, combat rules, and synchronization between the two players.

The server is authoritative over the game state. Instead of allowing each browser to independently decide the result of an attack, players send their actions to the server, which determines the outcome and sends the updated state back to both clients. This helps keep the game synchronized and prevents the two players from seeing different results.

We also use WebRTC for peer-to-peer camera communication between players, while the server handles the signaling required to establish the connection.

This gave us an architecture roughly like:

Webcam → Hand Recognition → Player Action → WebSocket → Game Server → Game State → Both Players

At the same time:

Player Camera ↔ WebRTC ↔ Opponent Camera

This architecture allowed us to combine real-time computer vision with a synchronized multiplayer game.

Challenges We Faced

One of our biggest challenges was combining computer vision with real-time gameplay.

Hand-sign recognition is not always perfect. Lighting, camera quality, hand position, movement, and the similarity between gestures can all affect detection. We needed to make the interaction feel responsive while preventing unstable predictions from accidentally triggering game actions.

Another challenge was multiplayer synchronization. In a real-time battle, actions from two players can arrive at almost the same time. We therefore needed the server to act as the authoritative source of truth and determine how simultaneous actions should interact.

Integrating multiple real-time technologies was also challenging. Our project combines webcam input, browser-side ML inference, WebSockets, WebRTC, frontend state management, and backend game logic. Each component worked differently, so connecting everything into one responsive game required us to carefully define the responsibilities of the client and server.

Finally, hackathon time constraints forced us to prioritize. We had many ideas for ninjutsu, animations, gestures, game mechanics, and visual effects, but we had to focus first on creating a complete playable multiplayer experience.

What We Learned

Building JutsuVerse taught us that creating a real-time application involves much more than making individual components work.

We learned how to run a computer vision model directly in the browser, process webcam input, and translate model predictions into meaningful game actions.

We also gained practical experience with WebSockets and server-authoritative multiplayer architecture. We learned why game state and combat resolution should be controlled by the server rather than trusting individual clients.

Working with WebRTC also helped us understand peer-to-peer communication and how signaling can be integrated into an existing multiplayer server.

Most importantly, we learned how different technologies can work together to create an experience centered around human interaction. Computer vision recognizes the gesture, WebSockets connect the game state, WebRTC connects the players visually, and the game itself gives those players a reason to interact.