super cool wizard guy

Inspiration

I remembered seeing a unique magic-casting game online and I wanted to show off how AI can unlock new ways of interacting with games

What it does

My app can detect when a user pairs the right hand gesture and keyword to cast spells, allowing them to accomplish goals in a 3d physics sandbox to solve puzzles and advance to the next level. There are 4 recognized spells, each corresponding to unique pose-word gestures

How we built it

I used a MediaPipe computer vision model to identify between hand gestures as well as an ElevenLabs api caller to transcribe speech and store the last word said, reliably sending spellcasting events to my Godot game project when the conditions are met.

Challenges we ran into

Bridging the MediaPipe/ElevenLabs script to the Godot project, managing scope of the project

Accomplishments that we're proud of

Its very interactive and uniquely fun, spells and levels are modular to allow for easy scaling

What we learned

Godot has built-in capabilities to receive signals from a UDP Websocket, MediaPipe gesture recognizer is largely plug-and-play, and multi-threading can be used to handle multiple persistent tasks at once

What's next for super cool wizard guy

Gameplay record/replay system so users can view their previous runs and compare their time/efficiency against other players

A wand to allow for fully mouse-and-keyboard-less interaction with the game world, maximizing immersion Full VR Integration

Built With

• elevenlabs
• godot
• mediapipe
• python