riemanns-roman-manifold

Inspiration

Sample libraries are a graveyard of good ideas. A producer with 50,000 samples spends more time searching than creating — scrolling through folders named kicks_final_v2, auditioning files one by one, hoping muscle memory fills in the gap where discovery should be.

The inspiration came from a simple question: what if you could navigate sound the way you navigate space? Acoustic similarity is geometric. Two sounds that feel alike live close together in embedding space. If you can reduce that space to three dimensions and put a camera in it, browsing becomes exploration.

What it does

Riemanns Roman Manifold turns your entire sample library into a navigable 3D universe. Every sound is a point in space — positioned by acoustic similarity, not folder structure. You fly through it in first-person (WASD + pointer lock), hover over points to see filenames, and click to play them back instantly.

Search is semantic: type "warm pad" or "punchy kick" and the camera flies to the closest matches. Drop in a reference sample and find everything that sounds like it. A minimap cube tracks your position in the global space at all times.

How we built it

Each audio file is processed through LAION-CLAP, a contrastive model trained on audio-text pairs that maps both audio and natural language into the same latent space — so a query and a matching sound end up geometrically close. The resulting $\mathbb{R}^{512}$ vectors are reduced to 3D with UMAP, preserving local neighborhood structure so similar sounds remain neighbors after projection.

Embeddings and coordinates are stored in PostgreSQL with pgvector. Search runs as cosine similarity:

$$\text{sim}(\mathbf{q}, \mathbf{v}_i) = \frac{\mathbf{q} \cdot \mathbf{v}_i}{|\mathbf{q}| |\mathbf{v}_i|}$$

Long tracks are split into stems first using Demucs (bass, drums, vocals, other), so a single song contributes multiple semantically distinct points. The backend is FastAPI

• uvicorn, running on an NVIDIA DGX Spark. The frontend is vanilla Three.js — no framework, no build step — served directly as static files.

Search triggers a three-phase animation: the camera rotates toward the result centroid (gimbal), matched points pulse with a radiating ring (illuminate), then the camera flies in (travel). A minimap cube tracks your global position throughout.

Challenges we ran into

Projection quality. UMAP is stochastic and sensitive to hyperparameters. Getting a layout where perceptually similar sounds cluster visually — without the projection introducing false neighbors — required careful tuning of n_neighbors and min_dist across multiple runs.

Long-form audio. Full tracks can't be embedded whole; CLAP expects short clips. Stem separation plus segmentation adds significant preprocessing time, so the pipeline had to be designed as resumable and cache-aware from the start.

Navigation without landmarks. A 3D space with no labels is disorienting. The crosshair-filename display, the illumination pulse, and the minimap were all solutions to the same problem: how do you know where you are when there are no streets?

Scale. Rendering 50,000+ points interactively while running async pgvector queries required decoupling the render loop from all I/O and managing draw calls carefully with per-vertex attributes over uniforms.

Accomplishments that we're proud of

The joint audio-text embedding space works better than expected — clusters that emerge from UMAP match intuitive sonic categories (kicks, pads, textures, vocals) with zero labels or manual tagging. Finding a sound by describing it in plain language and watching the camera fly to the right corner of the universe is genuinely surprising every time.

We're also proud of the standalone demo mode: when no backend is available, the UI generates a synthetic 6-cluster point cloud and runs the full gimbal → illuminate → travel animation with fake search results. The entire interface can be demoed from a static file server.

What we learned

• CLAP's shared audio-text embedding space is a remarkably strong prior for perceptual similarity, even on sounds far outside its training distribution.
• FPS navigation in a point cloud is a fundamentally better interface for large unstructured collections than any folder hierarchy.
• Three.js point clouds scale to 100k points when you treat the scene as a GPU data structure — instanced geometry, typed arrays, minimal state changes per frame.
• Dimensionality reduction is lossy by construction. The manifold is an approximation, and designing around that (rather than pretending it's exact) produces better UX.

What's next for riemanns-roman-manifold

• Structural segmentation: split stems into verse, chorus, bridge segments with boundaries snapped to the nearest downbeat before embedding — finer-grained points, more useful clusters.
• Metadata overlay: BPM, key, and roman numeral analysis surfaced as filterable HUD layers on top of the 3D space.
• ProducerPal integration: an Ableton Live AI assistant (Gemma 4 via Ollama, OSC bridge) will share the same shell, letting you go from finding a sound to using it in your session without leaving the interface.
• Audio-to-audio search: upload or record a clip directly in the browser and navigate to its nearest neighbors in real time.

Built With

• docker
• python
• pytorch