Track

Connectivity

Inspiration

Overwatch is a game where people get an array of characters to play. It is fun and incorporates huge numbers of fans all around the world.

One of our members, Haocheng "Bill" Yang, loves playing Overwatch. He especially likes to play the character Doomfist. However, Bill is facing a bottleneck--his competitive ranking cannot go up due to his bad mouse and keyboard skills. In fists of fury, he smashed his $200 Cherry MX keyboard and decided to make a change to this rather dull gaming world dominated by the controller, keyboard, and mouse.

We wonder if we can let Bill use his physical body to control the motion of Doomfist's skills in-game. So, Bill gathered three top hackers around him and set out to realize his seemingly impractical dream.

What it does

Using electromyography (EMG) sensors, we can make a sensor complex to collect and pre-process muscle movement information, pass it to our gaming laptop, and after post-processing reflect Bill’s muscle movement in-game as Doomfist’s movements.

How we built it

Hardware:

  • Solder one Myoware Muscle Sensor to the hook-up wires.
  • Find muscle using human anatomy knowledge and putting a load on the muscle.
  • Attach electrodes to muscles at the center of the muscle
  • Attach reference voltage point to a distant place on the skin
  • Test signal strength using an oscilloscope and adjust the gain on the sensors until the relaxed voltage is below 0.3 * 3.3 volts, and the excited voltage is above 0.7 * 3.3 volts.
  • Collect raw digital data and check accuracy--the result is bad with lots of type II errors, so we need to process the data via software
  • Use Tequila Nano microcontroller board to read analog voltages and send the data to the laptop -- success, but leads to a new threshold problem, that the physical connection of the system is not stable.
  • Reinforce structure and test
  • Idea verified, remove prototype from Bill Yang. This is the end of our first day.
  • On the second day, we rebuild the hardware structure. We abandoned the use of duct tape, cardboard, and electrical tape, and used more elegant solutions -- solder, screw, and hot glue.
  • We decided to add another function, which uses Arduino UNO alone with MPU6050 to sense the rotation of the helmet and send yaw, pitch, and roll data to the PC. It enables rotating the in-game first-person view
  • Add switch in Bill’s hand--enables walking
  • Hardware is finalized

Software:

Microcontroller side:

  • For the tequila nano, we will be programming with the Rath-HAL framework
  • with C language. We initialize the analog to digital converter (ADC) and the universal synchronous/asynchronous receiver/transmitter (USART) peripherals, and also the GPIO pins of them.
  • We sample the ADC with the shortest conversion period and read the digital input state of the button.
  • We combine them to a size-4 uint16_t array and send the data to the USART port. We will also send a newline character to distinguish the different packets.
  • For the Arduino, we build upon the MPU6050 example and send the roll, pitch, and yaw to the serial monitor.

PC side:

  • Use the Pyserial library to read serial data from USART ports connected with the microcontrollers. We use the built-in readline function to read the entire line.
  • Unpack the data from bytes to integers, and then normalize them to floating-point values which represent the relative muscle electrical voltage magnitudes between 0.0 and 1.0 and the yaw and pitch of the gyro. If the length is incorrect, we will just discard this packet (similar to UDP transmission).
  • Check whether the muscle electrical voltage exceeds the threshold value based on actual muscle electrical activity measured. We will also plot the data using matplotlib.
  • Perform specific keyboard actions using Windows API when the conditions are matched.
  • Perform 2-D mouse movement through post-processed gyro yaw and pitch using Windows API.

Challenges we ran into

Through the debugging process, we discovered that the electrical activity of the muscles measured from different EMG sensors sometimes overlaps with each other. For example, when our tester is performing a specific motion, multiple muscles (i.e. bicep and triangle muscle) exhibit electrical activity rising. We solved the issue by adjusting the thresholds of different conditions of different if statements in our programs through trials of testing.

In the design process, we found out that we have insufficient sensors to sense the leg movements of Bill, thus he cannot move in the natural way of just lifting his legs. However, we found out that by only moving forward, Bill is actually able to move around pretty easily by steering himself using his hat incorporated with a Gyro. Thus, we fitted a switch in Bill’s fist so that if he pressed the switch, his character would move forward.

Accomplishments that we're proud of

Our project connects sports and esports. It also transforms gaming into a more immersive experience through human-computer interaction between the game character and the gamer.

What we learned

On the software side, we practiced using several Python libraries to assist our development. We learned to use the Pyserial library to read serial data from USB ports. We also learned to use the Windows API to control the mouse and keyboard actions on our PC. During the debugging process, we also found the MatplotLib library useful for real-time graphing the muscle electrical activity from different EMG sensors.

What's next for Being Doomfist is better than playing Doomfist

We discovered a bright future of EMG-based input devices and will go on to make more generalized controllers able to operate in more precise, intensive tasks.

Also, one thing we can improve is to incorporate simple machine learning algorithms to increase the accuracy of our inputs.

Built With

  • emg-sensor
  • ergonomics
  • gaming
  • gd32v
  • hardware
  • human-computer-interaction
  • microcontroller
  • mpu6050
  • myoware
  • python
  • rath-hal
  • risc-v
Share this project:

Updates