One of our teammates Alice is a pro cello/piano player and she'd been stuck at her home for the stretches of lockdowns, and missed playing music with people or going out and exercising. So we had the idea to make a game where musicians, fitness freaks, and typemasters can battle each other!
What it does
BattleNotes is a web-based game. The player creates/joins a room and selects the character they wanna play - Musician, FitFreak, or TypeMaster. They wait until another player (opponent) joins the room.
Each character has their way of attacking/defending against the opponent. For example, FitFreak would attack a Musician by doing a few jumping jacks or push-ups, and the Musician could choose to defend by playing a tune given by the game engine. (Similarly, TypeMaster would attack/defend by typing a sequence of words in a time constraint and achieving a fast typing score) As the game goes on, the difficulty level gets harder and the competition intensifies- the Musician gets increasingly challenging and long melodies, and the FitFreak has to do more reps!
Attacks make opponents lose points whereas your defenses recover your points. When one of the players' HP gets to zero, they lose and the game is over.
How we built it
The mockups for the game were first designed using adobe illustrator and figma.
Musician The melodies are taken from the EsAC (Essen Associate Code and Folksong) Database, which contains over 7000 musical tunes, as well as the Weimar Jazz Database (nearly 500 jazz lead sheets). We categorize these into different “difficulty” levels based on their musical density and length, and extract the musical features. We use an obscure MIT Python library, in conjunction with a music engraving engine, to typeset the music and display it as an image for the musician to read. The recording of the musician’s playing is analyzed using FFT/spectrogram to detect beats and pitches and check them against the known correct sequence of notes they were supposed to play.
Typemaster The typing interface is based off of cheetoise, a typing tutor for python based on pygame. We get a corpus of words and sentences from various texts and joke sources and display them to the user to type out. The score is calculated as a function of accuracy, words per minute and length of sentence. This is then applied as an attack value or defense buff for incoming attacks.
Pushups The pushup detector interface is based off of an OpenCV and optical flow based pushup detection engine originaly authored by VNOpenAI. We take this engine and gameify it by using a retrained model to detect number of pushups and the detection threshold is dictated by the difficulty level. The number of pushups performed is the basis of the attack and defense values in the game.
game engine The basic game state is stored on mongodb and the engine itself is written as serverless functions.
Challenges we ran into
- Typesetting music programmatically, particularly from database formats we could find available, is surprisingly difficult, since music typesetting libraries are don't have the best documentation and have very little community support.
- Linking everything together.. it was an ambitious project!
Accomplishments that we're proud of
- making a fun game!
What we learned
- computational musicology tools
- ML for pushups
What's next for BattleNotes
- Improving audio detection (implementing neural network techniques to improve transcription), enable polyphonic detection
- Integrating other forms of challenges such as other exercises (squats, lunges, etc.) and more instruments
- Adding support for Apple Watch/Fitbit for tracking exercises more accurately.
- Turning this into an XBOX Kinect game