Keyboard Warrior

We present Keyboard Warrior: a simple fifteen-dollar boxing glove that can be used to control videogames through real-life boxing movement, using MediaPipe's human pose estimator and IMU-based gloves.

Comment 1

Inspiration

We felt inspired to make a hardware project because we noticed that in people in previous years tended to avoid hardware and focus solely on software applications, and we felt that in HackTrent there was a lot of unexplored opportunity for creative hardware projects like Keyboard Warrior. As for our core idea, we knew we wanted to make something fun and engaging, and one issue that bothered us was how expensive gaming gear has become over time. To address this, we came up with Keyboard Warrior, a simple and low-cost product for playing videogames like streetfighter or punch out through physical movement.

What it does

Keyboard Warrior is a system that lets you play videogames (with emphasis on fighting games) by shadowboxing. We created a pair of motion-tracked boxing gloves and used openCV to track the player's movements and joint angles. When a player throws a punch or moves around, our model detects what type of movement it is (jab, cross, uppercut, hook, slide left, slide right, block) and activates the corresponding keybind (j, i k, a, d, g, etc. these are customizable), effectively letting you play through movement.

How we built it

We used mediapipe's human pose estimator to track the player's joint angles from a webcam and create a skeleton of their movement. However, with only one camera, this pose estimation alone can't track 3D depth very well. So, we made gloves with an accelerometer and an esp32 with wifi and sent a regular stream of acceleration / gyroscope data to our computer, which when processed would help us determine what kind of punches the player throws. Now equipped with both the sensor data and the pose estimate, we created a random forest ML model to interpret the data. We recorded a lot of footage of us throwing different types of punches with the accelerometer gloves and then trained the model to predict different types of punches, and then finally correlated each punch to a customizable keybind. Eventually, we created a web dashboard to bring everything together. Voila! We could now play Streetfighter or Punchout by shadowboxing.

Challenges we ran into

This hackathon definitely was a learning curve for us. At one point, we fried one of our accelerometers which put us into a very tough spot since we only had 2 in our inventory. We were even going to improvise by using a phone as a boxing glove since it has an accelerometer in it. It was thanks to our mentor Bobby that we managed to find a replacement accelerometer and get on with the project.

Another issue we ran into is that our model was overfitting on one person's punches, however we managed to fix this by recording several volunteers and training on a more varied dataset. One benefit of using a random forest model is that it was very light and easy to train and retrain no matter how many times we wanted to iterate.

Accomplishments that we're proud of

  1. Reading the sensor data for the first time
  2. Getting keyboard output
  3. Figuring out how to use the phone's imu when our sensor initially got fried
  4. Controlling street fighter for the first time
  5. Seeing our model work after training

What we learned

  1. What a random forest model is and how it's effective
  2. How to use MediaPipe and OpenCV
  3. Soldering Through-Hole
  4. Communicating over wifi with UDP
  5. ALWAYS BRING EXTRA PARTSSSSS
  6. Test without power first so you dont fry your component
  7. Plan ahead so you have more time to sleep

What's next for Keyboard Warrior

  1. 3d printing a better casing
  2. Making it use lipo batteries, which make the controller even more compact
  3. Making it compatible with a wider range of IMUs and microcontrollers.
  4. Integrating with SteamOS so that we can run Keyboard Warrior on steam games.

Built With

Share this project:

Updates

Submission history