Enhancing Music Discovery and Engagement

Workflow of classification and recommendation
Workflow of New music ingestion
Sample Knowledge graph

Inspiration

While most of our ideas sprung from our experience of using the platform, we were left comparing the application to its competitors. Although TikTok was a first to the market in terms of short consumable content, there have been tech giants who have caught up with TikTok over the years. So figuring out specific gaps or areas of improvement was rather difficult, given the audience TikTok caters to.

The idea behind each of the phases of our solution is as below:

Emotion and Object Detection: The idea stemmed from the need to create a more engaging and interactive experience on TikTok. By analyzing emotions and objects within video frames, we can tailor music recommendations that resonate more deeply with the content and the user's mood. For instance, If a video is identified to have a joyful or energetic emotion, the system can recommend upbeat or high-energy music, creating a cohesive and enhanced viewing experience.
Ingestion and Classification of Music: This phase was inspired by the challenge of categorizing and organizing a vast array of music efficiently. By leveraging Mel Spectrograms and RNN classifiers, we can accurately identify and tag music genres, ensuring that the right music reaches the right audience. For instance, By accurately classifying music into genres, users can discover new music that aligns with their tastes, and emerging artists can be more easily discovered within their respective genres.
Biased Recommendation System: The inspiration for this phase came from the need to support and promote emerging artists who often struggle to gain visibility. By implementing a biased recommendation system, we can give these new talents a fair chance to be discovered and appreciated. For instance, The algorithm boosts new artists in recommendations, ensuring they appear more frequently in user feeds, thereby increasing their visibility and engagement.
Artist-Community Engagement: Building a vibrant artist-fan community was inspired by the need to create deeper connections and interactions between artists and their audiences. Incentivizing engagement through features like polls, sneak peeks, and collaborations encourages continuous interaction. For instance, An artist can host a poll about their next song release, engage directly with fans, and feature a "Fan of the Month" to incentivize active participation, thereby fostering a loyal and engaged community.

What it does

Our solution enhances music discovery and engagement on TikTok by:

Using emotion and object detection to tailor music recommendations based on video content.
Ingesting and classifying music through advanced ML techniques to accurately identify and categorize genres.
Implementing a biased recommendation system to promote new and emerging artists.
Encouraging artist-community engagement through interactive features like polls, sneak peeks, and collaborations.
Utilizing knowledge graphs to extend connections between new artists and their followers, fostering collaborations and content promotion.

How we built it

We built the solution through a multi-phase approach:

Emotion and Object Detection: Utilized computer vision techniques to analyze video frames and detect emotions and objects.
Ingestion and Classification of Music: Leveraged Mel Spectrograms and RNN classifiers to tag music genres and categories accurately.
Recommendation System: Developed a biased recommendation algorithm that prioritizes emerging artists, ensuring they gain more visibility.
Artist-Community Engagement: Integrated features to foster artist-fan interactions, like community polls and spotlight features.
Knowledge Graphs: Created knowledge graphs to map relationships between artists, their music, and user preferences, enhancing personalized recommendations and collaborations.

Challenges we ran into

Data Diversity: Ensuring a diverse range of emotions, objects, and music genres were accurately detected and classified.
Algorithm Bias: Balancing the recommendation system to promote new artists while maintaining user satisfaction.
Engagement Features: Designing features that genuinely incentivize and enhance artist-fan interactions without overwhelming the users.
Knowledge Graph Complexity: Building and maintaining comprehensive knowledge graphs to effectively map artist and user connections.

Accomplishments that we're proud of

Successfully implementing a multi-faceted recommendation system that boosts visibility for emerging artists.
Achieving high accuracy in emotion and object detection within video frames, enhancing the relevance of music recommendations.
Creating engaging community features that foster meaningful artist-fan interactions.
Effectively utilizing knowledge graphs to deepen connections and enhance personalized recommendations.

What we learned

The importance of diversity in training data to ensure robust and accurate ML models.
Balancing algorithmic bias to promote new talent without compromising user experience is crucial.
Interactive community features can significantly enhance user engagement and loyalty when designed thoughtfully.
Knowledge graphs are powerful tools for mapping and leveraging relationships between artists, music, and users.

What's next for Enhancing Music Discovery and Engagement

Expansion of Emotion Detection: Improve emotion detection capabilities to understand more nuanced emotions.
Enhanced Classification: Incorporate more sophisticated ML techniques to further refine music genre classification.
Community Features: Expand artist-community engagement tools, including more interactive and rewarding features for fans.
Integration with Other Platforms: Collaborate with music streaming services like Spotify, Apple Music, and Amazon Music for seamless data integration and broader reach.
Personalized Experiences: Continue refining personalized recommendations to create a more immersive and tailored user experience.
Knowledge Graph Expansion: Enhance and expand knowledge graphs to include more detailed and dynamic relationships between artists, music, and user preferences.