ClipForge

The clipping engine built for how India talks.

Comment

Inspiration

Every Indian creator sits on a goldmine they never mine: hours of long videos — podcasts, vlogs, streams — full of moments that would fly as Reels, buried under the grind of clipping, reframing, and captioning. The tools that promise to fix this (OpusClip and its clones) were built for English podcasts. Feed them Hinglish, where a sentence switches scripts mid-thought ("bro yeh setup insane hai"), and they mangle the captions, miss the real hook, and butcher Devanagari. India's creator economy is huge and served by tools that don't speak the way India talks. That gap is ClipForge.

What it does

Paste a YouTube link or upload footage, pick a clip style, duration, count, caption look, and aspect ratio — and ~90 seconds later you get a ranked set of vertical, captioned, ready-to-post clips. Captions pop word-by-word in sync with the voice, the speaker stays framed even when they're off to one side, every clip gets a title, and an editor lets you retrim, restyle, rename, and rewrite caption text before export.

How we built it

A single FastAPI service that also serves its own React + TypeScript + Tailwind frontend. The pipeline: yt-dlp/upload → ffmpeg audio → faster-whisper word-level timestamps → Claude (claude-opus-4-8) picks clips via a Hinglish-tuned virality prompt → ffmpeg renders with karaoke ASS captions. Three India-first touches:

  • Prosody-aware selection — audio energy \(E = \mathrm{clip}(100\,(0.55\ell + 0.30p + 0.15d),\,0,\,100)\) blended into the score as \(0.85\,s_{\text{LLM}} + 0.15E\).
  • Face-tracking reframe — OpenCV pans the crop to \(\mathrm{frac} = \mathrm{clip}!\big(\tfrac{f_x W_s - w/2}{W_s - w},0,1\big)\) so the speaker stays in shot.
  • Karaoke captions — one ASS event per word, popping in sync with the voice (6 styles incl. a saffron Desi Heat, plus no-captions).

Plus real auth + credits (PBKDF2, SQLite, server-enforced), 4 aspect ratios, and a glassmorphic UI.

Challenges we ran into

  • Editing captions without breaking karaoke sync — our first cut deleted the neighbouring line's edge word; fixed by assigning each word to one segment by its midpoint.
  • Aspect ratios — every caption margin and font had to scale with the canvas while keeping the default crop pixel-identical (\(\text{PSNR}=\infty\)).
  • Karaoke timing — overlapping ASS events stack instead of replacing, forcing strictly monotonic event boundaries.
  • Theme-reacting a JS-driven canvas background that reads colours as props, not CSS.

Accomplishments that we're proud of

  • A face-track reframe that turned a broken center-crop (speaker off-screen) into a perfectly framed short.
  • Word-synced karaoke captions that render Hinglish and Devanagari flawlessly.
  • Real, working economics: server-enforced credits with automatic refunds on failure.
  • Adversarial self-review that caught a silent credit leak, caption word-loss, and a scoring bug before users could.

What we learned

ASS/libass internals (BGR colours, \t animations, stacking events), that signal blends must be continuous (a 14-point cliff at \(E=0\) rewarded flat clips over lively ones), and that graceful degradation matters — with no Claude key, selection falls back to a density/keyword/energy heuristic so a job never hard-fails.

What's next for ClipForge

  • Multimodal analysis — an optional visual-understanding pass so it can catch silent moments (reactions, product reveals) the transcript can't see.
  • Hinglish virality scoring trained on what actually goes viral in India.
  • Scheduling & analytics — post to Reels/Shorts/TikTok and feed performance back into selection.
  • Payments and scale — Razorpay/Stripe credits and a queue-backed multi-instance backend for production.

Built With

Share this project:

Updates