Inspiration : A Picture Is Worth A Thousand Words (Songs)?
Our team wanted to connect to the intangible parts of the music listening experience.
We were inspired by the feeling of walking down the street on a particularly beautiful day and want some music to match the mood you're in. Do you ever feel like a picture is worth a thousand songs?
With the help of web frameworks, computer vision, and some music APIs, we were able to capture the essence of the experience. Audioscape captures the musical essence of your favorite scenery.
What it does
Audioscape captures the image that the user takes with their phone or mobile device. It then parses the image using the KMeans Algorithms for statistical analysis.
It uses the result of this analysis to create Spotify playlists matching the (mathematically decided!) "mood" of the picture.
How I built it
We split the major components into tasks
Challenges I ran into
Simon: I connected all the different parts of our app together to make them work cohesively. The challenging parts were to implement working login and file upload mechanisms that worked across both the front-end and back-end. There were several tutorials and guides online which were a bit over-engineered for the scope of our project, so I had to spend some time figuring out exactly which parts we would need and implement just those parts.
Stephan: I worked primarily on the mechanics of connecting the image values to song features, creating the playlist, and getting user data from the Spotify API. Authetication and understanding how the Spotify API returned information was difficult at first, but I was able to figure it out through lots of trial and error.
Lisette: While initially trying to build a python image analysis back-end using only the
Image libraries, it quickly became clear an elegant solution required turning the images in
numpy arrays, iterating through them, and finding means of every pixel RGB in the section (or cluster) and finding means inside each cluster. I needed to use a computer vision library, along with
scikit-learn and scikit image for image analysis.
Nicha: Challenges I ran into: I had a lot of trouble running the backend, so it was really hard to test whether the API requests correctly returned what was needed from the backend.
Accomplishments that I'm proud of
Lisette:I'm proud of being able to use a nuanced algorithm to accurately determine the "true mean" of the image from the user. Nicha: Accomplishments that I’m proud of: I am proud that I was able to figure about how to use React/HTML/CSS and design the frontend in such a short amount of time. I learnt React using a tutorial and this is the first real project I’ve done with it. Simon: I was happy to be able to connect all the parts so seamlessly, because usually that's what troubles me the most in my projects. It was nice to make sure that the frontend and backend could communicate with no weird bugs. Stephan: Being able to see a complete Spotify playlist after working on the API functions was very rewarding. I was also very surprised to see how closely those songs matched each other in terms of mood.
What I learned
Stephan: Working on a project like this is ultimately about adapting and good communication! Knowing that there will be things that are new to learn as well as having teammates you can ask for help were crucial for me in succeeding on my individual part.
Lisette: I learned about working with a successful developer team! Nicha: I learned to design a responsive front-end using React for both web apps and mobile apps. i learned how to connect fundamentals of web design to create a great user experience. Simon: This is the smoothest that a hackathon has gone for me and it was really because of how we all communicated well and clearly defined tasks for ourselves. In my future projects, I will try to replicate that and make sure we're not held up by uncertainty.
What's next for Audioscape
We'd love to develop a smoother front-end for the app - we think this could utimately tie in the experience of connecting music to beautiful images.