Tetrify: Tetris but with motion controls

Inspiration

All of us being game enthusiasts, we landed on the issue that many popular games require precise finger inputs, essentially creating an accessibility obstacle to playing such games. Mobility impairments in the hand can bar several hand motions such as clenching fists, pressing buttons with accuracy, and flipping switches, these being commonplace in gaming. Therefore, we decided on capturing and integrating the user's hand swipes into regular gameplay, using Tetris to present our idea.

What it does

Our project first calibrates the user's hand position based on their coordinates through the webcam. It then accordingly calculates which direction the user is shifting or rotating the block with their hand swiping movements to match regular player moves during a game of Tetris, effectively allowing those with mobility issues to play the game with its original functionality.

How we built it

Implementing computer vision with MediaPipe's hand landmarker model, we first averaged the user's hand position over the course of 5 seconds to calibrate the center position. Following the calibration sequence, we calculated a grid of directional ranges on the user's webcam to translate each of the user's hand swipes into a rotation or shift of a Tetris block, simultaneously displaying a Tetris game screen so that the user can continue playing the game until the current match completes.

Challenges we ran into

Our primary challenge consisted of linking the Tetris game implementation that we used, created in Pygame, to our hand tracking program with MediaPipe, as we needed to continuously iterate on it to accurately streamline the user's hand inputs to real-time gameplay.

Accomplishments that we're proud of

Despite all 4 of us being first-timers at a hackathon, we are proud of successfully completing the functionality of our Tetris hand tracking program in under 25 hours.

What we learned

Each of us became more proficient with MediaPipe and utilizing computer vision technology, all of us being new to the field. We also became more comfortable iterating upon existing codebases to match our specific needs.

What's next for Tetrify

Seeing as we can generalize the code and controls we made to many games, we want to make even more games accessible with our technology. At one point, we also want to make this a tool to allow people with mobility issues to control their entire computer more easily, and even be able to type and operate their computer just like anyone with perfect mobility. In the short term, we also want to make the calibration sequence more intuitive, and provide more instructions to the user so that they don’t need someone guiding them to use the software.