Originally, we intended to create a cubesat prototype to measure vegetation levels, but the obvious inability to actually launch the satellite, as well as the complexity of a more intense hardware hack discouraged us. When we arrived, in a last minute brainstorm, the idea came to us to create a project that converted images into music, and we decided to go for it.
What it does
IMG2MP3 captures an image using a raspberry pi camera, creates various metrics about prominent objects in the image, such as dominant color, intensity, saturation, and noisiness. It then uses these metrics to generate a few measures of music.
How we built it
We used the opencv computer vision library to generate our metrics from the captured image, though we had to design our own methodology to calculate the more complex metrics, like image noisiness. Those metrics were then passed into our own algorithm to generate melodic music, with each chord progression corresponding to an object in the image.
Challenges we ran into
Beyond the typical network communication problems and new tool growing pains typical of a hackathon, each of our team members had to catch up to speed in at least one part of the project. Although this somewhat slowed our development, it allowed all of us to learn new skills. The transition from laptop hardware to the raspberry pi also posed some issues, requiring a complete set up from scratch, as well as considerably slowing down our implementation.
Accomplishments that we're proud of
Some of our image processing algorithms were tricky but rewarding to implement, and our music creation routines were done with minimal library support, and create not unpleasant music.
What we learned
As a team, we learned about the considerable power of image processing, even without using any machine learning, as well as various techniques to accomplish it. Some of our team members learned to use git and github, and others learned something about music theory. Overall, the considerable cross-pollination available within this project, from computer vision to music theory to CAD, taught every member something about a skill they did not previously know much about.
What's next for IMG 2 MP3
We would like to see IMG2MP3 made into a more portable, continuously updating device, allowing the user to listen to the sights around them in real time.