Vibesort — Music Playlist Reordering Through Audio Embeddings
Inspiration
Most of us discover new music through social media these days, and YouTube has become a common denominator across age groups and devices for discovering and listening to music. We often come across a catchy tune, save it to a playlist, and intend to revisit it later. However, as our playlists grow, organizing hundreds of tracks by genre, mood, vocals, or instrumentation becomes increasingly tedious.
Wouldn't it be convenient to select a reference song and automatically arrange the rest of the playlist around its musical characteristics? Instead of manually searching through hundreds of saved tracks for something with a similar sound, why not let an algorithm find the closest matches and arrange them into a continuous listening experience?
Enter Vibesort: an application that reorders your playlists based on a selected reference track, helping maintain a consistent musical vibe with a single button press.
What it does
Vibesort uses pretrained music representation models to extract high-dimensional audio embeddings from songs and calculate their pairwise similarity in a shared latent space.
Given a reference track, the application constructs a weighted graph of all songs in the playlist and uses a greedy nearest-neighbor traversal algorithm to generate a reordered sequence.
Rather than simply sorting songs by genre, tempo, or popularity, Vibesort considers the learned audio characteristics of each track and selects consecutive songs based on their relative similarity.
The result is a playlist that attempts to minimize abrupt changes in musical characteristics while ensuring that every track is played exactly once.
How I built it
I developed the application around five main stages: audio embedding generation, pairwise distance calculation, adjacency matrix construction, greedy path traversal, and playlist reconstruction.
1. Audio Embedding Generation
Each song is processed using a pretrained music representation model, such as MERT (Music undERstanding Representation model), which extracts high-dimensional feature representations directly from audio.
The model generates contextualized feature vectors representing different portions of a track. These frame-level representations are aggregated through temporal pooling to obtain a fixed-dimensional embedding for each song.
For a playlist containing (N) songs, the resulting embedding collection is represented as:
$$ X = {x_1, x_2, \ldots, x_N} $$
where each song embedding captures learned musical characteristics such as timbre, instrumentation, rhythm, and other audio patterns.
These embeddings provide numerical representations of songs that can be compared mathematically, allowing the application to identify tracks with similar characteristics without relying exclusively on manually assigned genre labels.
2. Pairwise Distance Calculation
Once the embeddings are generated, the application calculates the similarity between every pair of songs using cosine similarity.
For two song embeddings (x_i) and (x_j), cosine similarity is defined as:
$$ \operatorname{sim}(x_i,x_j) = \frac{x_i \cdot x_j} {|x_i||x_j|} $$
The similarity score is converted into a distance metric:
$$ D_{ij} = 1 - \operatorname{sim}(x_i,x_j) $$
A smaller distance indicates that two songs have more similar learned audio representations, while a larger distance indicates greater dissimilarity.
This calculation is performed across every possible pair of tracks, producing a matrix of pairwise musical distances.
3. Adjacency Matrix Construction
The pairwise distances are organized into a weighted adjacency matrix:
$$ D = \begin{bmatrix} 0 & D_{12} & \cdots & D_{1N} \ D_{21} & 0 & \cdots & D_{2N} \ \vdots & \vdots & \ddots & \vdots \ D_{N1} & D_{N2} & \cdots & 0 \ \end{bmatrix} $$
Each row and column corresponds to a song in the original playlist. An entry (D_{ij}) represents the distance, or transition cost, between songs (i) and (j).
Because cosine distance is symmetric, the resulting matrix is also symmetric:
$$ D_{ij} = D_{ji} $$
The diagonal contains zeros because the distance between a song and itself is zero.
The matrix effectively represents the playlist as a fully connected, weighted graph. Every song is a vertex, and every possible transition between two songs is an edge weighted by its calculated distance.
Precomputing this matrix allows the application to reuse the same pairwise distance information when experimenting with different starting songs, without regenerating embeddings or recalculating similarities.
4. Greedy Path Traversal
To generate a reordered playlist, Vibesort employs a greedy nearest-neighbor traversal algorithm.
The user selects a starting song, which serves as the initial vertex in the graph. From that point onward, the algorithm repeatedly selects the nearest unvisited song relative to the current song.
The traversal follows these steps:
- Initialize the current vertex using the user's selected starting song.
- Mark the current song as visited.
- Examine the distances between the current song and all unvisited songs.
- Select the unvisited song with the smallest distance.
- Append the selected song to the reordered playlist.
- Set the selected song as the new current vertex.
- Repeat until every song has been visited.
Mathematically, the next song is selected using:
$$ v_{\text{next}} = \underset{v_j \in U}{\arg\min} \ D_{\text{current},j} $$
where (U) represents the set of unvisited songs.
The resulting sequence forms a Hamiltonian path through the playlist graph: every song is visited exactly once, and each consecutive pair is selected to minimize its immediate transition distance.
This is particularly important because the algorithm does not simply rank every song by its similarity to the initial reference track. Instead, it continuously evaluates the next transition relative to the previously selected song, producing a chain of locally similar tracks.
5. Computational Complexity
For a playlist containing (N) songs, each represented by a (d)-dimensional embedding, the computational costs are:
| Operation | Time complexity | Space complexity |
|---|---|---|
| Embedding generation | Model-dependent | (O(Nd)) |
| Pairwise distance calculation | (O(N^2d)) | (O(N^2)) |
| Adjacency matrix storage | — | (O(N^2)) |
| Greedy path traversal | (O(N^2)) | (O(N)) |
The pairwise distance matrix is computed once and reused during traversal. This makes it possible to experiment with different starting songs without repeating the computationally expensive embedding and distance-generation stages.
6. Result
The final output is a reordered playlist that attempts to minimize abrupt changes in the learned musical characteristics between consecutive tracks.
By combining pretrained audio representations with graph-based traversal, Vibesort provides a practical approach to playlist sequencing that goes beyond conventional sorting and recommendation techniques.
The architecture also provides a foundation for future improvements, including multi-objective transition scoring, genre and mood weighting, and more advanced graph optimization algorithms.
Challenges I ran into
One of the biggest challenges was optimizing the processing pipeline to reduce the time required to retrieve audio samples and convert them into embeddings. I had to balance audio resolution and bitrate against processing time and data consumption to avoid unnecessary overhead.
I accelerated audio retrieval through multithreading and concurrency, maximizing network bandwidth utilization for I/O-intensive tasks.
Another challenge was encountering HTTP 429 errors caused by YouTube request rate limits. I experimented with authenticated session cookies and had to account for access restrictions and request throttling while developing the retrieval pipeline.
Finally, iOS introduced additional restrictions around embedded video players, making it difficult to provide seamless music playback through a web application. Handling differences in playback behavior across browsers and devices required additional architectural considerations.
Accomplishments that I'm proud of
I optimized the processing architecture sufficiently to generate an adjacency matrix for hundreds of music videos within the duration of a single track, typically a few minutes.
I was also able to leverage the MERT-v2 model's learned audio representations to capture meaningful similarities between tracks. These representations provided a useful foundation for grouping songs with related musical characteristics and generating more coherent playlist sequences.
Most importantly, I brought together audio representation learning, vector similarity calculations, and graph traversal into a functional application that can reorganize an existing playlist around a user-selected reference track.
What I learned
I learned that high-dimensional vector representations, while powerful for identifying similarities, can consume considerable computational resources and memory, particularly when calculating pairwise distances across an entire playlist.
I also learned the importance of distinguishing local optimization from global optimization. My current implementation uses a greedy heuristic that selects the closest available next track at every step. Although computationally efficient, this approach does not guarantee the globally shortest possible playlist sequence.
Finding the globally optimal ordering corresponds to a fixed-start, open-ended Traveling Salesperson Problem (TSP) variant, which is computationally difficult for large playlists (EXP time).
This helped me understand the trade-offs between computational efficiency, scalability, and optimization quality when applying graph algorithms to real-world music applications.
What's next for Vibesort
I envision expanding Vibesort beyond YouTube to support other major music services, including Spotify, Apple Music, and Amazon Music.
The current implementation already supports Spotify playback, providing a foundation for extending playlist analysis and reordering across multiple music platforms. Each provider introduces different API capabilities, playback restrictions, and subscription requirements, which will need to be handled through platform-specific integrations.
I also see opportunities to improve the sequencing algorithm itself. Some potential directions include:
- Multi-objective optimization: Incorporating genre, mood, tempo, energy, instrumentation, and other musical attributes into a configurable transition cost function.
- Clustering: Using algorithms such as K-means to identify groups of related songs before determining their ordering.
- Global path optimization: Exploring approaches such as 2-opt, simulated annealing, and other combinatorial optimization techniques to improve upon the greedy traversal.
- Transition-aware embeddings: Analyzing track intros and outros independently to evaluate how well the ending of one song complements the beginning of another.
The current version is just a demo for a hackathon, but it can truly evolve into an everyday tool used by disk jockeys and lay people alike.