Inspiration
This game was inspired by a game called Snake Wave. Snake wave is a based on the classic game of Snake but with a twist: it uses finger tracking to control the snake. We really liked the idea of building on a game by incorporating a different, arguably more difficult, set of controls. This lead us to create Goose Pong: a spin on the classic game of Pong.
What it does
Goose Pong uses computer vision to track the head movements of players to control the paddles that hit the ball. Unlike Pong, the paddles have free range of motion on half of the screen rather than just moving up and down. We also incorporated head rotations in the game to add another element of skill expression.
The ideal setup for the game would be to have two people standing about a metre away from the computer. Each person will stick to their own half of the screen and try to hit the ball into the opposing player's goal. The first to 5 points wins!
We also added some settings that could make the game easier or harder for various skill levels. Players can adjust the size of the goals, the speed of the ball, and the size of the paddles.
How we built it
We used Pygame to create the game loop and the various menus in the game. To track faces, we used a computer vision model provided by Google's Media Pipe Solutions and used OpenCV to read input from the computer's camera.
To handle menus, we used polymorphism to define a reusable interface for menus, each with load, update, and render functions. We can easily store these subclasses with a dictionary and key into that dictionary to switch between menus. For the game entities, like the paddle, ball, and goals, we used an object-oriented style to encapsulate all of the data and logic required to make these entities function properly.
To map the paddles onto the face movements, we used the model to identify the eyes of the player. We then created a vector based on the two points and applied a rotational matrix to it allow the paddles to rotate along with the face. We also added an error margin for the facial detection to prevent the paddles from jiggling too much because the model keeps micro-adjusting the points on the face.
Challenges we ran into
There were two main challenges we faced in creating the game. The biggest challenge was getting collision detection to work properly. Because we allowed rotation of the paddles and had two moving parts, it was difficult to get the collision to work perfectly. This was especially true when the ball or paddles moved at high speeds. We originally worked with a collision algorithm which would detect if the ball intersects with the rectangular paddle hitbox at each frame, but we quickly realized that if the paddle or the ball moves too fast within frames, then the collision may not be detected, especially since the face tracking limited our application to 30 fps. We then moved on to a collision algorithm better suited for detecting these types of collisions called the Swept AABB algorithm. However, we ran into complications with this algorithm as well because the Swept AABB is easy to implement when only one object is moving, though we have two. In the end, we had to devise an algorithm which could handle the fact that there were two moving objects which we wanted to detect the collision, but even this algorithm is not perfect and many shortcuts were made to keep the algorithm simple. It works by detecting whether the ball is on opposing sides of the paddle in the current frame compared to the frame before and uses this to decide if a collision occurred. In hindsight, I think we should have taken advantage of a library like Pymunk, which is a 2D physics engine that might be able to handle collision between moving objects.
Another challenge we faced was getting the computer vision model to recognize faces effectively and mapping the face movements onto the paddles. At first, we used a model from the OpenCV library but it was not very good. We tried to process the camera input in different ways such as changing the colour and trying out different computer vision models to detect the faces. After lots of trial and error, we found that combining RGB colouring with Google's Media Pipe Face Detection model was quite effective.
Accomplishments that we're proud of
We are proud of having a finished product that works reasonably well. This project had a lot of moving parts and we are quite fortunate to have all of them working well together. On the more technical side, we are very pleased with the accuracy with which the paddles map to head movements. When we first started the project, we thought that would be one of the most challenging parts, which it was, but it turned out well.
What we learned
We learned a lot about game development including how the game loop functions and how to organize the different menus in a game. We also had more practice using Numpy to manipulate arrays to perform calculations for game physics. Lastly, we had the chance to experiment with different computer vision solutions which might be useful if we want to continue along with the trend and create more games with CV motion controls.
What's next for Goose Pong
We plan to make a single-player game mode where the player tries to sort different kinds of waste into the gain points. We could also add skins or different character and background designs so that each game feels like a fresh new experience. If we are feeling ambitious, we could even try to design a server-client architecture which can allow players to play on multiple devices.
Log in or sign up for Devpost to join the conversation.