Inspiration- We wanted to recreate an online Otamatone experience and spread the fun of this rather unique instrument.

What it does- Uses computer vision to identify mouth and hand movements in real-time. We utilize their hand movement to control the pitch and mouth movements to control whether any sound is outputted. We then map data from midi files of songs and match the two data points and generate an accuracy rating, creating a scoring system. Overall it is basically a GuitarHero-styled game.

Challenges we ran into - We found that loading OpenCV opening on the HTML of the page to be very weird because of the lack of support on OpenCV with the JS wrapper class.

Accomplishments that we're proud of - We were able to create a MIDI file to WAV file converter. We were also able to establish computer vision that recognizes hand and mouth movement in realtime. We are also proud that we were able to utilize this information to affect the movement of the objects on the screen.

What we learned - We learned a lot about implementing OpenCV, tone.js & Phaser.

What's next for Otamahero - We need to package our application and also add more cool features.

Share this project: