Kinetic Keys

Body Tracking

Inspiration

We believed it would be a fun challenge to play the games that we love in a creative and non-traditional manner. Many things in our daily lives require nimble usage of our hands—but would if we couldn’t use them? Could we play a game, e.g. Tetris, using other parts of our body? What would be easy and fun to use without requiring too many resources or setup?

What it does

Our project reads in the user’s actions from their webcam. Specific movements will activate and mouse inputs that can be reconfigured using the included interactive GUI. The inputs can then be used to play games or interact with other digital content

How we built it

We utilized OpenCV to read in the user’s webcam as input and then mapped their body to a skeleton using Google’s MediaPipe, which was lightweight and able to be run in real time. We preprogrammed certain actions, such as arm or leg raising, based on the relative positions and angles of the joints, which were then fed to Pynput mouse and keyboard inputs. A customizable GUI was then constructed using Tkinter, which allows the user to configure the key bindings in the manner that best suits their needs.

Challenges we ran into

One of the biggest challenges was making sure that the program would not mistakenly do other inputs or spam one input constantly when the user was neutral. In fact, bug fixing ended up taking up the bulk of our time spent. We solved these challenges by adding a 0.3 second cooldown to inputs which prevented inputs from being spammed. We also defined clear thresholds for movements to be considered valid, and allowed users to enable/disable specific movements which prevents unintended inputs from movements they don't want to use.

Accomplishments that we're proud of

The program is actually quite reliable and accurate when it comes to translating body movement to inputs on the computer. Tetris was quite playable and enjoyable and we didn't think we'd actually be able to pull this off within such a limited amount of time.

What we learned

We learned a lot about using a wide range of technologies, including how to analyze poses using mediapipe’s pose detection system and how to process and learn how to process real-time video using OpenCV. We also learned how to debug and fix issues regarding input detection and body tracking.

What's next for Kinetic Keys

For Kinetic Keys, we want to decrease the latency between measuring body inputs to keyboard inputs to make playing more ping-sensitive games easier. We hope to expand Kinetic Keys’s user base and make it something that players will seek out to use.