Gester - A Gesture Game Library and Visual Arcade (Table 83)

Gester - Platform / Arcade
Game Gallery
Pong w/ Team
2048
I need the max win!
BOXES!
Fancy!
Pikachu!

Inspiration

When deciding what we wanted to do for the hackathon, we knew we wanted to build a game around using gestures. However, we struggled to come up with both a library to do so with and an idea that we all liked. That's when we decided to focus on building a platform with the goal of allowing others to build gesture-based games. Specifically, we wanted to streamline this process through a game library so that building games is quick and efficient.

What it does

Game Library

Our game library serves as a PyGame wrapper around our Gesture Library in order to serve developers a seamless experience for integrating gesture input into games. We found PyGame to be feature lacking and overly verbose in its requirements for getting a game off the ground. As such, we have expedited the experience of developing games through our custom game design architecture.

Gesture Library

By adapting a specific section of Google's early release MediaPipe, Hand Landmarkers, we are able to (relatively) easily fetch the coordinates of hand joints the model recognizes on camera. We then detailed gestures and specifications for those gestures to present an easy API to interested developers. Rather than have to configure and wrap a model themselves, they can plug-and-play by giving a single callback to our library.

Arcade

In order to prove the efficiency of our gesture based game library, we built several games that show off many different features. These were created to showcase the power of the platform and aimed to maximize personal enjoyment.

Pong - The first game built shows off the most basic features of the game library: basic shape rendering, collision detection, and most importantly gesture support. The user battles the computer in the classic game from the 70s while controlling their paddle with their hand.
2048 - In this seemingly simple math puzzle game, the player is able to slide around powers of 2 by wagging their finger from side to side. Through enough trial and error they are able to achieve the 2048 tile.
Simon Says - Simon Says demonstrates the adaptability of our gesture recognition. The player is tasked with showing off their knowledge of various hand signs by demonstrating them in front of Simon when asked.
Box Stacker - In this game you are granted control of a platform (through your hand) and must control it to catch and stack boxes. Watch out though; the box-a-pocalypse is near.
Casino Simulation - Our game engine offers texture and sound loading. This is demonstrated by the thrilling experience of running the slots.
Pikachu Tomogachi - A game where you can feed Pikachu berries as he wanders around in a field. Lots and lots and lots of berries... A tranquil, peaceful, calm, and otherworldly experience.

How we built it

All parts of the project were written in Python. We used PyEnv for environment manager and Poetry for dependencies. Our gesture library wrapped MediaPipe and utilized OpenCV for video processing. The game library took advantage of PyGame. Any arcade games were built solely with our own library.

Challenges we ran into

Setting up our environment and Poetry was difficult as none of us had worked in a large team on a large Python project before. Luckily, having background in NPM/Cargo helped.
MediaPipe was difficult to get up and running. There isn't a ton of documentation around the product and thus we had to do a lot of experimentation. Furthermore, it isn't as featured as we expected and needed to write an entire library in order to ship the input to the game library.
We used a GitHub submodule for the separate gesture library built to support the game library and this was difficult due to our unfamiliarity with submodules.
Architecting the system and trying to create our API ahead of time was challenging due to our uncertainty about the end product.

Accomplishments that we're proud of

Creation of a game interface on top of MediaPipe and PyGame.
A series of fun games that show off individual features of the library.
Staying up all night.

What we learned

Although python is famous for its ease of use, it is unmistakably a feature filled language with a large amount of depth. Developing a library for the use for others required many high level decisions that needed to use many advanced features of python. While developing games we also required a create deal of creativity in order get our games working with a library that was not battle-tested.