mAHHHHrio

Inspiration

We share a passion for old games and were excited to have the opportunity to breathe new life into Mario. Additionally, the active element of motion control could easily be used in schools or day cares to make physical activity more fun.

What it does

mAHHHrio allows you to play the original Mario game with motion controls. Run right and left by actually running right and left, and jump by raising your arms high up or jumping in the air.

How we built it

We used an AI model to recognizes poses and crucial points (such as knees, elbows, and hands) through OpenCV and our computer's webcam. This information was processed and combined with a Python implementation of Mario to allow for motion based control of the game.

Challenges we ran into

We abandoned another project which proved impossible and began building mAAAHrio around 2 am this morning. Coordinating all the components of this game in real time proved an effective programming challenge. For instance, in order to display image data and read the webcam in a performant way, we needed to run the input and output in parallel threads, passing data in between. Additionally, we needed to strike a delicate balance between input resolution and speed to make the game fun to play and responsive.

Accomplishments that we're proud of

I consider every stage of this project a big milestone. Reading in webcam data and classifying joints and "skeleton" structure using AI is interesting on it's own, but combining it with a real time game like Mario in a playable and interesting way involved many performance challenges. We began with many of the graphics and some of the basics of a Mario emulator, but ultimately had to reverse engineer and refactor nearly the entire application.

What we learned

We learned how to leverage AI for optical input, specifically for full body detection. We also learned a lot about reverse engineering applications built by others through refactoring lots of our Mario game.

I also learned pretty much the whole layout to Mario's first level :)

What's next for mAHHHHrio

Ideally this code could be optimized and exposed to play any platformer with motion based controls. Personally, I think that a live action Donkey Kong or Street Fighter would be super fun. Packaging up and open-sourcing this project would allow anyone with a webcam and a decent computer to play these games on most PCs or maybe raspberry pi home arcades.

This kind of accessibility means that it could be used for education, as well. Motion based games like Pokemon Go or games involving the balance board and WiiMote have proven that active games are an effective way to help kids (and young-at-heart adults) enjoy healthy physical activity. Mario is great, but there's no reason the same approach couldn't be used to teach fitness classes or physical therapy in a better format.