MP Fighter

• 

  leap into action

• 

  fight your friends

• 

  creator your own unique character

• 

  choose your fighter from your saved collection

Inspiration

Vision tracking software has been available for some time, but is complex to set up for many applications. Google's MediaPipe library seemed like an interesting tool to attempt a vision project with.

Another inspiration came from past experience competing in game jams. During game jams, animation has always been the hardest element to get up to a quality standard. Motion tracking seemed to have strong potential as a solution to this problem.

What it does

This project records the motions of a user acting out moves for a fighting game. These moves consist of hit boxes and damage areas, which we generate by applying motion tracking AI to a camera input. The AI also masks out the background, so the camera frames can be used as game sprites. This allows users to quickly record themselves to create custom characters. All the characters are saved to files allowing you to face off against your friends.

How we built it

Image processing is done using Googles MediaPipe Pose API for motion tracking and OpenCV2 for reading the camera feed. This data is made available on a locally run Flask web server. The server is used by the game engine to request the processed vision data and masked images. The game has a full character creator UI built it, which serves as the interface over the vision processing.

Challenges we ran into

• Integrating Flask with OpenCV2 was complicated as most OpenCV2 examples assume a single thread, while Flask requests may come from many.
• Complete recored characters were large (~150 MB) so we had to be careful not to load too many into memory at once.
• The data structure all the character data had to be carefully mapped so the game could animate through it efficiently.

Accomplishments that we're proud of

• Integrating the output of a vision processing system into a game.
• Creating a full fighting using this vision data for all collision detection.
• Definining a consistent format for storing move animations and collision data.

What we learned

• How to use Google's Media Pipe API and OpenCV2.
• How to communicate between Python and a game engine using a webserver.
• How to animate collision shapes over time in Godot.

What's next for MP Fighter

We plan to use the vision technology developed for this project in more complex game projects in the future. We're also interested in applying the captured motion data to a 2D skeletal animation.

Built With

• flsak
• gdscript
• godot
• mediapipe
• opencv2
• python