Music swap

Working with creators who create content for brands, a common issue is when they post content that has licensed music. Music swap makes replacing on-brand and on-content music as easy as 1,2,3.

Comment

Summary

Creators working with brands can't post sponsored content with trending copyrighted music, and swapping the track by hand takes hours and rarely fits the edit. Music Swap takes a reel (upload or link), measures the edit, watches and listens to it, then composes three original, rights-free instrumentals timed exactly to its cuts and key moments. It recommends the best fit for the brand and learns from every pick.

The problem and its impact

Brand deals are a core income stream for creators, and music rights are where they break. Platforms mute or block sponsored posts that use copyrighted tracks. Agencies spend editing time re-scoring clips with stock music that ignores the cut points. Music Swap turns that into about 80 seconds: same video, same voice, new music that lands on the edit. It remembers each brand's taste, so the tenth video for a brand is better than the first.

How it works

  1. Upload a 3–90s video or paste a social link, and pick the brand.
  2. A Daytona sandbox measures the edit: scene cuts and keyframes with ffmpeg; tempo, beats and energy peaks with librosa.
  3. Gemini watches the full video (tone, pacing, what's said, timestamped beats) and writes producer notes on the original music (groove, bass, instruments, key, energy arc), without the melody or song name.
  4. Claude Sonnet reads all of it plus the brand's memory and writes a brief and three composition plans whose sections change exactly at the cuts: Faithful, On-brand and Wildcard.
  5. ElevenLabs Music v2.5 composes the three tracks. Each gets a lead-in and tail that are trimmed off, so the music starts mid-groove and ends on a resolve.
  6. If someone is talking, ElevenLabs voice isolation keeps their voice and drops the copyrighted song, and the new music is ducked under the speech. Every preview is mastered to social loudness (-14 LUFS).
  7. Two judges score each option: Claude against the brief and brand, Gemini by actually watching and listening to the finished preview. The top score is recommended; the creator decides.
  8. The creator's pick is saved as a new fact in the brand's memory, superseding older preferences.

Architecture

Next.js 16 + TypeScript with shadcn/ui, one app for the UI and API. Timing comes from DSP, not the LLM: scene cuts always win when placing section boundaries. Media work runs in an isolated Daytona sandbox per job. Optional services (memory, tracing, voice, listening judge) degrade gracefully instead of failing a job. The UI streams results as they arrive: insights, then plans, then previews, then scores. 55 unit tests cover timing, parsing, mixing and scoring.

Sponsor tools and how we used them

  • Daytona: one isolated sandbox per job runs all media processing: ffmpeg scene detection, keyframes and audio extraction, librosa tempo and energy analysis, and the final mix and mastering of each preview.
  • ElevenLabs: Music v2.5 composes three original instrumentals from cut-aligned composition plans. Voice isolation keeps the creator's speech while removing the copyrighted track. Its upload copyright scan also flagged our test reel's original song, confirming the problem we solve.
  • Arize: OpenTelemetry tracing of the whole pipeline. Each job is one session tagged with its brand, Claude calls show as LLM spans with full prompts and token counts, and both judges' scores are logged as eval attributes.
  • SurrealDB Agent Memory: per-brand scopes holding typed, time-aware music preferences. New picks supersede old ones, and the analyst and both judges read the same memory.
  • Anchor: a cloud browser opens the social link and captures the video for import, with a yt-dlp fallback.
  • Also used: Claude Sonnet 5.5 (analyst and plan judge), Gemini (video and listening notes, listening judge).

Challenges

  • ElevenLabs refuses copyrighted audio as a style reference, so we carry the original's feel through Gemini's producer notes instead of copying it.
  • The voice isolator keeps any human voice, including a singer-songwriter's vocals. Gemini now confirms someone is actually talking before we keep a voice.
  • Instagram preloads the next reel, so grabbing "the largest video on the page" sometimes imported someone else's clip.

What's next

Let Gemini's timestamped beats (a punchline, a reveal) become section boundaries alongside scene cuts, add a copyright check on the input, and offer per-brand sound profiles that agencies can share across creators.

Built With

Share this project:

Updates

Submission history