-
-
Query results in EnsembleAtlas
Inspiration
Applications like Shazam and even Google allow users to search for music just by humming, playing, or typing the name of the song. However, some people may want multiple similar songs with a certain melody. Or they want to identify songs by their feel or what's going on in the song. This is what EnsembleAtlas is for.
What it does
EnsembleAtlas enables users to search a song with either natural language describing the song or by uploading an audio file of a sample from it, whether it be played, hummed or produced anyhow. It will match the query to the catalog of songs built and return the best matching songs, along with timestamps for where those sweet moments the user requested are.
On the backend, one can ingest and add songs to the catalog. This starts several worker processes, enabling for an efficient distribution of work. If a worker dies, then the others will absorb their work.
How I built it
This was built in Python, and the LAION CLAP model for text and audio embedding was used for inference. Codex drove the development process.
Challenges I ran into
Shared CLAP model vs. CLAP model per worker - I was running into performance issues attempting to make all the workers share the same CLAP model. I decided that for the current use case, it's better to have each worker use its own copy of the CLAP model. It reduces performance overhead at the cost of extra memory. Optimizations for this are planned for the future.
Accomplishments that I'm proud of
This is my first project related to distributed machine learning, an area I've wanted to specialize in.
What we learned
I gained a deeper understanding of distributed ML and distributed performance optimization while integrating one of my favorite hobbies (music).
What's next for EnsembleAtlas
There are a lot of ways to extend EnsembleAtlas. Some examples:
- Distributed training
- Optimizing use of shared CLAP model
- Physical distribution
- Allowing users to record audio for query
Built With
- clap
- codex
- python
