Musical Analysis - Generative AI playlists (MusicBuddy)

Inspiration

Our project draws inspiration from various music, gaming, and community engagement sources. We have seen many players and users trying to find videos and music from their childhood, especially in Play Station 2). This platform includes various aspects and motivations, such as nostalgia moments, iconic music, and heroics.

What it does

Our web app contains functional requirements such as login/sign up and uploading music and videos. Our features incorporate GenAI analysis to predict the mood of the music and media, integrating Spotify API data and manual data. Our service offers a recommendation system based on the user's listening history and mood preference. With the advanced AI generation, users will have a better experience everywhere, no matter whether they are going to the gym, studying, or walking; our website offers a comprehensive and adaptive music curation.

How we built it

We used Angular for the front end to display the user profile, such as mood, listening history, music generation, and playlist creation. For the backend system, we used Bun/Hono because it is super fast and high-performance, which manages CRUD operation for user data and mood preference, and database management system. For machine learning, we used our machine learning built from scratch for training and analyzing audio in the music, and videos. After that, it will predict the mood based on the metadata of the media files. For authentication and authorization, and business logic , we used Amplify, Lambda, and API Gateway along with AWS Cognito to build whole system of the backend, with custom Bun runtime. The storage contains 2 places, S3 Bucket and AWS RDS. The S3 contains audio upload from user, RDS manage application data such as user's data, history, and playlist lists.

Here is the flow of the application

The user signing up and login to the system. Then, they will choose the music they like (the music will be generated randomly) which contains various mood such as sad, happiness, or neutral, and old school music). After that, the data will be sent to the backend through API Gateway and reach to the server where Bun is managing, then it will be save to the database. Besides that, the server will send the data to machine learning for mood prediction based on user selection. After analyzing, the user will get the initial mood of their profile. Another flow of the application is when user want to upload the music or video. After submitting their file, it then will be sent to machine learning to retrain the model based on the music, it then will added to user's profile as the author of the music.

Challenges we ran into

Challenges we ran into was setting up the AWS environment and Bun/Hono code base. We encountered versioning issues because one member does not have correct version of Node of Apple Silicon device. Then we also ran into the problem setting up the API Gateway. Our project was done yet, the backend is done and finished with training machine learning recommendation system but the frontend still missing the music generation features. The frontend problem we encountered is the node version, which took us 2 and 30 hours to fix that

Accomplishments that we're proud of

Accomplishments we are proud of is we finally testing our machine learning algorithms and our AWS environment worked probably. We also managed to complete 50% of the frontend Angular. And one of my teammates learned a lot of AWS integration to the application and deep dive into Angular.