Hands Off My Planet!

Inspiration

Most games still reduce powerful actions to a button press. We wanted to explore a more natural and expressive way to play: what if your hands were the controller?

That question became Hands Off My Planet!, a gesture-powered alien combat game set on a hostile planet. We wanted every spell to feel like something the player physically performs.

We also wanted the player to feel less alone in this unfamiliar world. That inspired us to create a voice AI companion that observes the current game state, warns the player about threats, and fights alongside them as a guide.

What it does

Hands Off My Planet! places the player on an alien world under attack.

Using a webcam, the game recognizes the player's hand gestures and translates them into combat abilities. Players can:

Launch energy attacks with hand gestures
Interact with alien technology and environmental objects
Ask a voice AI companion for context-aware combat advice
Review battle summary for future improvement

The AI companion is connected to the current game state. It can understand information such as the enemy's condition, the player's available abilities, and the objective of the current encounter.

Our main interaction highlight is the ability to purely use hand gesture for battle system. Through the physical interaction, the player can stay engaged with the game world.

No controller is required. The player's hands become the weapon, while the AI companion becomes their voice in the unknown.

How we built it

We developed the project with Cursor and divided the system into three connected layers:

Gesture recognition
The webcam feed is processed using python and media pipe. Hand landmarks are classified into a small set of gestures.
Game system
The game was built using unreal engine. Recognized gestures are converted into game events that trigger attacks, defensive abilities and visual effects.
Voice AI companion
We integrated the Grok Voice API to create a responsive in-game companion. The game provides the AI with structured information about the current situation, allowing it to explain enemy behavior and offer short, relevant tactical suggestions.

The gesture system handles immediate combat locally, while the AI handles dialogue and guidance. This separation keeps the game responsive even when a voice response takes longer to generate.

Challenges we ran into

One major challenge was making gesture recognition feel like a game mechanic instead of a technical demonstration. A gesture could be correctly recognized and still feel frustrating if it activated too early, repeated unintentionally, or produced unclear feedback.

Also, for dubhacks specifically, it's hard to set a feasible project rank for the team to achieve the technology completion and the whole interactivity set within 24 hours.

We addressed this by:

Choosing a small number of visually distinct gestures
Requiring gestures to remain stable briefly before activation
Showing the currently recognized action on screen
Providing immediate visual feedback when an ability activates
Designing fallback controls for testing and live demonstrations
Frequent team discussion and brainstorm
Clear division of work

Another challenge was connecting generative voice AI to real-time gameplay without allowing network latency to interrupt combat. We kept combat actions local and used Grok for short, contextual dialogue at meaningful moments.

Finally, we had to balance technical ambition with a short hackathon schedule. We focused on one polished combat experience instead of building many incomplete levels.

Accomplishments that we're proud of

We are proud of combining two different interaction modes:

Hands for fast, expressive action, for best interactivity
Voice AI for guidance, strategy, and companionship

Together, they create an experience that feels more natural and immersive without requiring a traditional controller.

What we learned

We learned that successful gesture interaction depends on understanding player intent, not only recognizing a technically correct pose. Clear feedback, forgiving timing, and simple gestures matter as much as model accuracy. That understanding is crucial as we deepen our understanding about user map later.

We also learned that AI works best in a game when it has a clearly defined role. Our AI companion does not replace the player or control the core gameplay. It observes the game state and provides information that helps the player make decisions.