Grapheoke

Inspiration

We are passionate about math and enjoy hacking for fun. After seeing art and animation others have created using Desmos, we wanted to push its limits and see if we could use Desmos to make music.

What it does

Grapheoke displays two graphs: one for the lyrics and one to play the music for a given song. These two are synchronized to create a karaoke machine.

How we built it

For the lyric part, we chose a font using a TTF file. The TTF glyphs (characters) are represented as Bezier curves, which can be converted to parametric functions that are either linear or quadratic in our case. We used the ttf_parser library in Rust to read the TTF file and extract the Bezier curves, and then converted them to a format we could programmatically input into Desmos. For the audio, we first reverse engineered Demos' source code to extend the range of notes (the default is limited to one octave). We then used open source MIDI libraries to turn them into mathematical functions that we then programmed with our specific parameters to display and play on the graph. The audio utilizes Desmos' Audio Trace function to "play" a graph. By plotting the graph at specific points, we can create music. The two were implemented together using Rust, which called a Python script that wrote the HTML file to display our final output. JavaScript was used to synchronize the playback so the audio and lyrics played at the same time.

Challenges we ran into

Different fonts take different amounts of time to load, so depending on the font we used, the synchronization would be slightly different each time. Furthermore, MIDI software natively fixes the playback to be synchronized; however, we were working with raw MIDI data, so the synchronization proved to be a challenge, as we had to implement it through JavaScript manually.

Accomplishments that we're proud of

We were able to work together complete the project and get both the lyrics and audio working at the same time. We weren't sure that both graphs could be displayed and manipulated independently, and we are proud that we managed to show both.

What we learned

We learned how to reverse engineer obfuscated JavaScript. We also learned how TTF and MIDI files are represented and how to work with them.

What's next for Grapheoke

The synchronization can still be polished, and we want to implement multi-language support with better Unicode rendering and cubic Bezier curves. We would also want to have a more user-friendly interface so anyone can use Grapheoke.