We were inspired by Steven Universe Soundtrack Attack, which has classic guitar-hero style rhythm gameplay with what appears to be a platformer playing in the background; the platformer seemed to be manually made, but we wanted to be able to computationally create levels for a rhythm game.
What it does
The game spawns enemies at times based on different sounds in the song.
How we built it
We used Python to get data about the amplitudes of a wave file in the time domain, split the song into equal-length segments of half a second, and applied an unsupervised clustering machine learning algorithm to group
Challenges we ran into
Reading the amplitudes of a wave file is difficult; we needed to convert from bits. Initially, we tried training the learning algorithm based on a large collection of songs; processing these songs into a format for our learning algorithm took upwards of two hours. Afterwards, we had 3 million training samples but found that song segments would most often be grouped based on similarities within the songs and not differences between individual sounds in a song; first, we attempted to solve this through normalizing the data, but because of the long time required to process songs, we switched to clustering the sound segments based only on the single input song. We also initially had too short segments, resulting in impossibly fast spawning elements; the segment length was adjusted from 40 milliseconds to 0.5 seconds. We had been clustering segments based on the magnitude of the power spectrum found with the Fourier transform of each segment; however, we found that simply using the magnitude of the amplitudes worked much better for identifying significant differences in sounds over time. Some songs would result in very uneven clusters, resulting in too many or too few enemies spawning. We attempted solving this by trying different clustering algorithms with different parameters, There were unexpected problems with appropriate collision behavior.
Accomplishments that we're proud of
The game visibly responds to the rhythm of upbeat songs.
What we learned
One member of the group had experience with audio processing and machine learning, while the other had experience with game development; thus, we each learned valuable skills from one another. We also learned pros and cons of different clustering algorithms.
What's next for RG&RG's Rhythm Game: Rover Go!
We'd like to fix collisions in the game, add winning conditions, add powerups, and increase the accuracy of signal processing.