cutback

Cutback is a local-first AI video editor that finds repeated takes, explains which delivery is strongest, assembles the cut, and exports the result without uploading the creator's footage.

Comment

Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

What we learned## About Cutback

Cutback is a local-first, client-side talking-head video editor designed to take the tedious friction out of rough cuts. By processing everything locally in the browser—from audio extraction and speech-to-text to take comparison and final rendering—it gives creators the speed of cloud-based AI tools without sacrificing data privacy or forcing heavy desktop software installations.

Inspiration

Editing talking-head videos—whether for tutorials, vlogs, or presentations—usually involves hours of scrubbing through timelines, finding flubbed lines, hunting down alternate takes, and cutting out awkward pauses. Existing tools either require uploading large video files to external cloud servers (raising privacy concerns and bandwidth bottlenecks) or rely on clunky, desktop-bound legacy software.

Cutback was built to answer a simple question: Can we run a complete, intelligent video editing assistant entirely client-side in the browser, keeping 100% of the user's media private while automating the most repetitive parts of editing?

How It Was Built

Cutback is engineered as a modern web application leveraging browser-native media APIs and client-side machine learning:

  • Core Stack: Built with TypeScript, Vite, Web Audio API, and a custom orchestrator (app.js) wiring pure logic modules to a responsive UI.
  • Speech-to-Text: Runs Transformers.js inside a dedicated Web Worker (transcription.worker.ts) to execute Whisper completely off the main thread, keeping the UI silky-smooth during heavy inference.
  • Timeline & State Management: Implements bounded undo/redo history, deterministic take ranking using text embeddings, local draft recovery, and real-time synchronized preview.
  • Rendering & Export: Uses client-side WebM recording (with MP4 fallback paths) to generate and download the final edited video directly from the browser's memory.

Key Challenges & Technical Hurdles

Building a fully functional video editor inside a web browser came with distinct engineering roadblocks, particularly around machine learning inference and automated editing logic:

1. Tuning Lightweight Whisper Models for the Browser

Running speech-to-text in the browser means managing strict memory and performance budgets. Initial attempts using standard model variants either bogged down the main thread or failed to load entirely.

  • The Fix: We experimented extensively with compact English Whisper models via Transformers.js, offloading the heavy lifting to a background Web Worker. This ensured the app could cache the model locally for instant subsequent loads while maintaining responsive frame rates on the UI.

2. Overcoming the "Pause" Transcription Trap

During early testing, the transcription pipeline would frequently stall or completely stop transcribing if the speaker paused for too long or if there were prolonged periods of ambient silence between takes.

  • The Fix: We restructured the audio chunking and Web Audio decoding pipeline. By implementing robust VAD (Voice Activity Detection) logic and ensuring graceful fallback boundaries, the worker could safely step over silent gaps without hanging or dropping downstream timestamps.

3. Improving Accuracy & Take Alignment

Whisper outputs raw text with timestamps, but matching alternate reads of the same sentence required more than just literal string matching. Slips of the tongue, filler words ("um", "uh"), and slight phrasing variations caused the algorithm to misalign alternate takes.

  • The Fix: We introduced transcript-connected local embeddings and repeated-take grouping. By analyzing semantic similarity vectors between alternate reads, the engine learned to reliably cluster variations of the same thought:

$$sim(\mathbf{u}, \mathbf{v}) = \frac{\mathbf{u} \cdot \mathbf{v}}{\Vert{}\mathbf{u}\Vert{} \Vert{}\mathbf{v}\Vert{}}$$

This allowed Cutback to accurately group flubs alongside the final good take regardless of minor wording differences.

4. Solving the "Wrong Cut" Problem

Early iterations of the auto-cutter frequently struggled to find the exact boundaries of clips to remove, occasionally cutting off the tail end of a good sentence or leaving awkward half-seconds of dead air.

  • The Fix: We built an explainable ranking algorithm combined with bounded trims and manual take overrides. Instead of blindly trusting a black-box model, Cutback surfaces the ranking logic visually in the UI, allowing creators to inspect why a take was grouped, audit the before/after impact summary, and easily override or adjust cut boundaries with full undo/redo safety.

What Was Learned

Building Cutback provided deep insights into the limits and capabilities of modern web browsers as application runtimes. Managing heavy binary video buffers, coordinating multi-threaded Web Workers for machine learning inference, and designing a bulletproof local-first state recovery system highlighted the power of treating the browser as a true native application environment.

What's next for cutback

Built With

Share this project:

Updates

Submission history