Inspiration
We wanted to create a fun and interactive way to combine artificial intelligence, computer vision, and music discovery. Since people often express themselves through their clothing and accessories, we wondered whether AI could use someone's visual style to guess their music taste.
That idea led to Spot-A-Vibe, a web application that turns a webcam snapshot into a personalized music vibe. Instead of answering a questionnaire, users simply step in front of their camera and let AI guess their musical identity. We wanted to make AI feel more approachable by turning it into an entertaining experience that could also help people discover new music.
What it does
Spot-A-Vibe uses a person's clothing and accessories to predict their music genre and sub-genre. When someone steps in front of the camera, the application detects their presence, starts a countdown, and captures a snapshot. Users can also capture a photo manually.
The AI predicts one of nine music genres: Pop, Rock, Metal, Rap, R&B, Country, Electronic, EDM, or Jazz. It then identifies a sub-genre within the predicted category, producing results such as Rap ยท Trap.
Users can view confidence scores and ranked probability breakdowns for all nine genres and the five sub-genres associated with their top result. If they connect their Spotify account, Spot-A-Vibe recommends songs that match their predicted vibe and allows them to save those recommendations as a Spotify playlist.
How we built it
We built Spot-A-Vibe using Python, Flask, JavaScript, and several machine learning technologies.
For person detection, we used OpenCV.js and a MobileNet-SSD object detector running in the browser. This allows the application to detect when someone enters the camera's view without continuously sending video frames to a server. Only captured snapshots are uploaded for genre analysis.
For music genre prediction, we integrated Cloudflare's Clef-Flash vision-language model. The model analyzes clothing and accessories through a multiple-choice classification process. We use two inference passes: the first predicts the most likely genre from nine options, and the second predicts a sub-genre from five options within the selected genre.
We developed the backend with Flask to process snapshots and return the model's predictions and probability scores. For deployment, we separated the web server from the GPU-powered Hugging Face Inference Endpoint, allowing the application to serve web requests without requiring a GPU on the web server itself.
Finally, we integrated the Spotify Web API to recommend songs based on the predicted genre and sub-genre probabilities. Users can connect their accounts, browse recommended tracks, and save their selections as private Spotify playlists.
Challenges we ran into
Integrating multiple technologies into one cohesive experience presented additional challenges. We needed browser-based person detection, automatic snapshots, model inference, confidence scores, Spotify authentication, and playlist creation to work together smoothly. We also had to account for camera permissions, Spotify redirect configuration, and the time required to initialize the model.
Another challenge was designing a meaningful classification system for something as subjective as music taste. Clothing does not reliably determine what someone listens to, so we approached the project as a playful AI prediction rather than a definitive assessment of a person's preferences. We also designed sub-genre descriptions to account for visual style as well as musical identity, helping the model make more specific predictions.
Accomplishments that we're proud of
We are proud of turning an unconventional idea into a functional AI-powered music discovery experience. Spot-A-Vibe brings together computer vision, a vision-language model, a web backend, and a music streaming API in one application.
We are particularly proud of implementing automatic person detection and snapshot capture, making the experience interactive without requiring users to repeatedly operate the camera. We also went beyond a basic genre prediction by including sub-genre classification, probability breakdowns, and Spotify song recommendations.
Another accomplishment is our deployment architecture. By separating the lightweight web server from GPU-based inference, we made it possible to host the application without requiring dedicated GPU resources for every web request. Supporting scale-to-zero on the inference endpoint also helps limit GPU costs when the application is not being used.
What we learned
Building Spot-A-Vibe helped us gain practical experience integrating computer vision, multimodal AI, backend development, and third-party APIs. We learned how to use a pretrained vision-language model for classification, process model outputs into meaningful results, and connect those results to a music recommendation experience.
We also learned how important infrastructure decisions are when working with large AI models. Model size, startup time, GPU memory, and deployment costs all affect how an application performs in practice. Separating inference from the web server helped us better understand how AI applications can be structured for deployment.
Most importantly, we learned that building an AI-powered application involves more than getting a model to produce an answer. The surrounding interface, data flow, user experience, and limitations of the model are just as important in creating something useful and engaging.
What's next for Spot-A-Vibe
We want to continue improving Spot-A-Vibe's accuracy, usability, and music discovery features. One area we would like to explore is expanding the range of visual styles and sub-genres the application can recognize while refining the prompts and classification process.
We would also like to improve the recommendation system by offering more diverse song selections and giving users greater control over the music that ends up in their playlists. Additional features could include saving past vibe results, comparing predictions across different outfits, and sharing results with friends.
Ultimately, we want to keep experimenting with ways to make AI-powered music discovery more interactive, personalized, and fun. Spot-A-Vibe is a starting point for exploring how visual AI can connect self-expression with music in unexpected ways.
Log in or sign up for Devpost to join the conversation.