Inspiration
We were inspired by the theme of HackGT this year, and we wanted to see how we could apply some really interesting software tools to gaming. We were really interested in computer vision and hand tracking software, so we wanted to combine those ideas together to create our project.
What it does
Our project can play games like Stardew Valley, Pac-Man, and Tetris in really cool and new ways. Instead of using a boring old keyboard and mouse, you can upgrade to the future of controllers, your very own hands! You can control your character's movement by just moving your hands around in view of your camera, and use different hand gestures and hand movements to perform different actions in the game.
How we built it
We used libraries like OpenCV, mediaPipe, and pydirectinput as part of our project. OpenCV allowed us to use our camera and collect image data that we fed into mediaPipe functions to be able to track different parts of the user's hands with great accuracy. After we were able to track the gamer's hands, we wanted to find a way to map those gestures into inputs into the game. So pydirect input was used to simulate using keyboard inputs through python. So the player can move their hands around, OpenCV and mediaPipe can track their hands, and pydirectinput can use that info to control the active game.
Challenges we ran into
We had a few issues getting our hand tracking software working with the keyboard interrupt API. If we wanted our character to move smoothly in the game, we would sometimes run into issues where our hand tracking software would start to stutter and lag, and this would cause some very un-smooth game movement. We also had trouble deciding what different hand gestures and movements we should use to correspond to different game inputs, while still making the hand movements seem intuitive and easy to use.
Accomplishments that we're proud of
We were able to use multithreading to fix the timing issues between the keyboard interrupt API and the hand tracking software, and how games can run a lot smoother and provide a better gaming experience. We also managed to use mediaPipe's hand tracking software with multiple hands, so that we can make intuitive controls to allow the user to perform even more actions in game.
What we learned
We learned a lot about computer vision, primarily OpenCV and mediaPipe, as well as general hand tracking tips and tricks and we also learned how to simulate keyboard and mouse inputs. We also learned a lot about tensorflow since we had experimented with some tensorflow gesture identification models, even though we unfortunately were unable to fit them into our project.
What's next for Gesture Controlled Gaming
I think we could expand to even more actions and allow for a wider range of usable gestures, maybe we could even incorporate some hand gesture identification models in the future. I think it would also be cool to expand our capabilities to even more games, personally I think this would be a really cool idea for something like a racing game. Or maybe we could create a GUI so that the user can map their own custom hand movements to game inputs.
The track we want to primarily focus on is the endlessly entertaining one. If we are able to be considered for multiple tracks, then we would like to be considered for the Upgrade and Retro Hacks also.
Log in or sign up for Devpost to join the conversation.