Hand Gesture Car Racing

What it does

Our game is a racing game that requires no additional driving equipment or keyboard input. Using just your hands and computer webcam, we created a fully functional driving vehicle that can race across a course.

For the main driving mechanism, we split the driving into two parts: driving and steering. In order to control the driving speed, the user must move their hands closer (faster) and farther (slower). To make sure the driving is always comfortable, the user must drive in a "thumbs up" position, and can zero the velocity at any time by putting their thumbs down.

We created a race track to accompany the car, with a fully functional lap timer and checkpoint system, where the entire lap won't be counted until the car reaches all of the checkpoints and the finish line. Additionally, if the car leaves the track, the speed will drastically slow down.

Finally, we added a settings menu to fine tune any multiplier to the user's preference, including velocity and turning multipliers. We also created a stats menu in order for the user to visualize how their car is moving.

How we built it

Our two tasks were essentially computer vision, and the game itself.

For computer vision, we decided to use Google's open source mediapipe "Gesture Recognizer" model, ensuring that we'd have the opportunity to extract gesture data. With some somewhat straightforward math, we're able to calculate the centers of the hands and determine the angle that the hands make relative to the origin. Additionally, we used the proportion that the distance is directly proportional to 1 / Area ^ 2, allowing us to directly relate the velocity to the (relative) distance that the hands were from the screen

For the game, we decided to go with PixiJS, an HTML rendering engine for 2D games. We decided to create a top down racing game due to simplicity, and made the background move relative to the car, rather than the car move around a background. This allowed for bigger maps and a more natural feel to the driving. We used node.js and express.js to host the website via Heroku.

Challenges we ran into

Our biggest challenge was definitely getting the driving mechanism to completely run with the movement feeling natural. The car would often fly out of the screen randomly, or the car would turn opposite to the direction we wanted to, and so on and so forth.

Additionally, setting up the Gesture Recognizer model and allowing it to connect with multiple hands provided a little more of a challenge than we expected. However, we were able to eventually implement the model in the end.

Accomplishments that we're proud of

I'm extremely proud of the fact that the driving in the game, once you get the hang of it, is surprisingly natural and fluid. It's impressive that we were able to capture such a solid driving experience using only a webcam and hands.

Additionally, it was difficult to complete all the game logic, and completing it was a task that we are proud of.

We were also able to clean up the code and make it neat and readable, which proved to be a huge timesaver during troubleshooting.

What we learned

We learned how to integrate computer vision into a webapp, and learning how to seemlessly integrate two different applicatoins into one big app. Additionally, we learned to be diligent with Git, and make sure we stay commiting to keep a solid commit history, in the case we want to look back in previous iterations of our code.

What's next for Hand Gesture Racing Game

In the future, it's possible to add a leaderboard and login system, providing an ability to log and see "top times" of a track. Additionally, adding a multi-track selector would be another goal, allowing the user to pick and choose their favorite maps. Maybe we could even implement a track creator with this system. We also can add more complex driving mechanics, including drifting and a more sophisticated acceleration/braking system. For our settings, we could implement a cache system to remember the previous settings.