Eclipse

An AI producer in your iMessage group chat. Drop a song, say “we gotta sample this,” and get drum-free chops, a video of where they sit, and flips built on real chord progressions.

Comment

Inspiration

A friend sent me a song in iMessage with one line: "I'm about to sample this." What came next is the routine every producer knows: Shazam, Google, WhoSampled, then dragging the file into a DAW to find the part worth chopping. The conversation was in the chat, but all the work happened somewhere else.

Eclipse puts a producer in the chat. The name fits the theme twice: in an eclipse the moon passes in front of the sun, the way a sample hides one record inside another, and the word itself contains "clips."

What it does

Eclipse is an AI agent that lives in a real iMessage group chat through Photon Spectrum.

  • Chops on the beat. Send it a song (audio or video) and say "we gotta sample this." It tracks the beat, finds the "1" of every bar, and sends about six one-bar chops with the drums removed: melodic phrases, loops, and hits, each labeled with the chord it sits on.
  • A highlight video plays every chop in place in the song, so you hear where it came from.
  • Flips that progress. Pick the chops you like and Eclipse suggests orders whose chords form a real progression and loop back home, then asks before building. On one test track it proposed Bm–D, C#m, Em–B, E, which opens on the tonic and pulls back into Bm every time around. Or type your own order ("1 3 2 4", "1 2 2 4") and it builds that 8-bar flip.
  • Plain-language edits. "Start 3 a beat earlier." "Make 4 two bars." "Try them 2 semitones lower."
  • Knows songs from links. Share a Spotify or Apple Music link and it knows the song. Ask what it samples and it researches the answer on the web and names its sources.
  • Knows when not to talk. "This is hard" and "what time is dinner" get no reply.

How I built it

  • Agent: TypeScript on Photon Spectrum for iMessage. Claude reads the conversation and returns a structured decision: whether to speak, what the person wants, and which action to run.
  • Audio: its own TypeScript DSP, with ffmpeg only for decoding and encoding. STFT, harmonic–percussive separation to remove drums, dynamic-programming beat tracking, downbeat detection from chord changes, kicks and snares, Krumhansl–Schmuckler key estimation, and chord recognition per half bar.
  • Arranging: every possible order of the picked chops is scored on common-practice root motion (falling fifths, steps, cadences into the tonic) and on how smoothly the audio of one chop flows into the next.
  • Video: drawn frame by frame with @napi-rs/canvas and piped into ffmpeg.
  • Links and research: songs are identified with the iTunes lookup API and Spotify's Web API, and sample history comes from Claude with its web search tool.
  • The rule we kept: algorithms find the evidence and Claude reasons over it. Captions and suggestions come only from measured numbers.
  • Testing: everything is checked against synthetic songs with known answers. We built it pair-programming with Claude Code.

Challenges I ran into

  • Half a beat of drift. Our first tempo estimate was 117.5 BPM on a 118 BPM song. Over two minutes that drifted half a beat, so chops started mid-bar and nothing fit together. Refining tempo to 0.02 BPM and adding beat tracking brought the grid's jitter down to about half a millisecond.
  • Kicks leaking through drum removal. Standard harmonic–percussive separation let kicks through because a kick filled most of the analysis window. A longer window just for the bass range cut the leftover drum energy from 23% to 3% in our tests.
  • Arrangements no producer would make. Our first flips shuffled slices, and the next used fixed patterns. A producer's feedback pushed us to harmony: Eclipse now reads each chop's chords, orders them into real progressions, and asks before building.
  • iMessage and Windows details. A video that arrived inside a "group" message was invisible to the agent until we handled it, and ffmpeg's 1.5-second launch time on Windows forced us to batch every encode (15 s down to 5 s per job).

Accomplishments that I'm proud of

  • A full music-analysis pipeline in plain TypeScript, with no Python and no ML model: beat and downbeat tracking, drum removal, key and chord recognition.
  • On synthetic songs with known answers it gets the tempo within 0.1 BPM, puts every downbeat on the true bar start, and recognizes every chord.
  • It behaves like a person in the chat: it stays quiet when it should, asks before it builds, and says so when it can't tell something.

What I learned

  • Timing is everything. A 0.4% tempo error was the difference between chops that fit and chops that don't.
  • Producers think in whole chops and chord progressions, not slices.
  • Asking beats guessing. Suggesting an order and letting the producer choose made the flips better.
  • Streaming links don't carry audio, so the right move is to identify the song and ask for the file, not to rip it.

What's next for Eclipse

  • Finding what a recording samples from its audio alone, by fingerprint and chroma matching against candidate sources. That was the original goal.
  • Sample lineage trees, from the source record to the flip.
  • Neural stem separation for cleaner drum removal, and drums or 808s under the flips.
  • A web view of chops, flips, and lineage.

Built With

Share this project:

Updates