ALAN (Artistic Luminance Audio Network)

Homepage
About the Team Page
Sign In / Sign Out Page
Code

Inspiration

With the growing popularity in machine learning and image processing, we were inspired to pursue a project relating to transforming classic images and photographs into melodious tunes. Coming from musical backgrounds, with all the members in our team being part of organized symphonies for the greater part of our lives, we immediately turned to making a project relating to music once the theme of RidgeHacks was announced to be art. Lastly, we all enjoy volunteering and aiding the growth of our community, with all four of us making booths at our school’s Sensory Fair to expand the knowledge of those born with disabilities, leading us to make an application targeted towards aiding those who could utilize our assistance.

What it does

Our application takes in a user-inputted image and runs it through our intense, deep cognition machine learning algorithms to transform the image into text, which is then transformed into music based on certain features of the image, such as mood, color theory, and music theory. Additionally, we utilize user-authentication and verification software to maintain security on our application and protect our users from exploitation.

How we built it

The first part that was trained and then integrated were the machine learning models. We have two machine learning models. The first machine learning model reads the image and outputs the mood of the image (e.g. triumphant, peaceful, noisy, fiery) as well as a description of what the image is about. This text is then put into another machine learning model, which takes the description of the image and it’s mood and makes inferences about what music would complement it. Once these models were completed, we created REST API’s which allows us to call the model on-demand and take the output. Then, we worked heavily on the front end. We used Next.js in order to style the website, but we decided that it was important to create infrastructure in a way that would scale easily. For this reason, we implemented user authentication with Clerk. After this, we thought more of how the software could be more usable. Therefore, we integrated Cloudinary, a content distribution network service, into our code, which allows the user to easily upload photos to ALAN. Finally, we connected everything together. This involved bridging the frontend to the backend by hosting it on Vercel.

Challenges we ran into

Training the model ate up a decent amount of our time, as we had to develop our own CNN using TensorFlow. This led to a lot of problems with time management. Additionally, although we had known that integrating the frontend and backend, we underestimated just how much time this bridging would take.

Accomplishments that we're proud of

We are proud of being able to divide the workload very evenly. Mohit worked on training the machine learning models and bridging between frontend and backend, Aritra worked on the REST API’s, Malav worked on the ability to upload images to the server, and Arav worked on the general GUI of the website. The team was able to get a lot done this way, even though there was a lot of work involved.

What we learned

Our team learned multiple different frameworks and libraries like Framer motion, Shadcn, express.js, REST API’s, and Cloudinary. The team learned to not over-commit and use our personal skills to our advantage.

What's next for ALAN (Artistic Luminance Audio Network)

From a community standpoint, we hope to use ALAN to integrate it into the lives of everyday people who could utilize it to better improve their lifestyles. ALAN is hopefully going to be used by visually impaired folks who, despite their disability, are able to appreciate art and images by transforming their necessary senses for appreciating art from sight to hearing.

From a technical standpoint, we want to add more instruments that users can pick between (unfortunately, we weren’t able to train more instruments due to time constraints). Additionally, we would like to add a database that stores the images that previous users have uploaded so that we can store a library of harmonious music that users can listen to. Finally, we want to try branching out from just images and try generating music that can go along with other types of media, such as videos.