Voice Remix

Inspiration

I wanted to build a demo where someone could edit a song or music clip just by speaking. I also wanted the result to stay visual, so the user could understand the change and adjust it manually afterwards.

AI music generators are good at creating a first version, while traditional DAWs give precise control but require a lot of production knowledge. Voice Remix explores the space between them: talk naturally, inspect the exact proposed edit, and keep control of the timeline.

What it does

Voice Remix is a voice-first visual multitrack arranger. A user can speak or type a request such as:

Move the final chorus four bars earlier and make the drums harder, but keep the bass unchanged.

The app turns that request into a Music Diff that shows every proposed operation, its before-and-after value, assumptions, and any protected tracks. The user can compare Current and Proposed audio, remove individual operations, Apply or Discard the edit, and use Undo and Redo afterwards.

The working demo also supports local full-song import, individual stem replacement, synchronized multi-stem import, source-derived waveforms, manual stem mute controls, a guided judge mode, and stereo WAV export of the committed arrangement.

How I built it

The editor is built with React, TypeScript, Tone.js, Canvas waveform rendering, and Zod.

GPT-5.6 Sol receives compact project context through the OpenAI Responses API and returns Structured Outputs constrained to the supported music-edit schema. The model never receives raw audio and never mutates the project directly. Server-side deterministic code validates section and track IDs, clamps values, protects named tracks, and creates a versioned transaction for the UI to review.

The OpenAI Realtime API provides live voice transcription, conversational replies, and editor tools for transport and editing actions. Audio decoding, playback, waveform analysis, and export run in the browser.

How I used Codex and GPT-5.6

Codex was my build partner throughout the project: technical research, PRD and architecture, the multitrack audio engine, Music Diff transaction model, voice interaction, screenshot-driven UI debugging, regression tests, deployment checks, and the reproducible demo-video pipeline. I committed frequently so the repository shows how the implementation evolved.

GPT-5.6 Sol is part of the working product rather than only the development process. It translates contextual creative intent into a validated edit plan that the user can inspect and selectively apply.

Challenges

The hardest part was keeping the AI flexible without allowing it to invent project state or silently change audio. Structured output solves syntax, but it does not provide domain trust. Voice Remix therefore separates proposal from execution: the model proposes, deterministic code validates, and the user decides.

Synchronizing real stems, keeping Current and Proposed playback mutually exclusive, implementing partial acceptance, and making section moves audible and reversible were also major engineering challenges.

What I learned

For a creative AI tool, showing what will change is as important as understanding the request. Non-destructive proposals, protected tracks, and reversible history make the AI feel more like a collaborator than an autopilot.

What's next

Next steps include automatic section analysis for imported songs, draggable edits that use the same transaction model, persistent projects, collaboration, and semantic selective revert.

Built With

• codex
• gpt-5.6-sol
• openai
• openai-realtime-api
• openai-responses-api
• react
• structured-outputs
• tone.js
• typescript
• vercel
• zod