HEJA

Inspiration

Inspiration

In April I came home from a week in Miami and the Bahamas with 250 drone clips — 52 GB, 63.6 minutes of 4K. Six months later I had edited exactly none of them.

That's not a motivation problem. The hard part of editing isn't cutting; it's watching. Before you can make a single decision you have to sit through an hour of footage, remember which take was the sharp one, and hold a story in your head. The actual cutting is the easy part.

So I went looking for a tool. Everything I found did one of two things: dumped my clips into a beat-synced highlight reel I couldn't change, or handed me a timeline and left the watching to me. A thread I found from another traveller summed it up — half a year of footage, two memory cards, and no tool anyone could recommend.

The gap was obvious: nothing watches your footage, agrees on a story with you, and then hands the decisions back.

What it does

vlogai is an assistant editor, not an auto-editor.

1. Import — point it at a folder. It reads capture time, GPS, duration and orientation, groups clips into days, splits long takes into shots, and scores every shot for focus and exposure so the soft and blown-out ones are flagged up front.
2. Log — the AI watches every shot (keyframes plus any transcribed speech) and writes a log entry: what happens, who's in it, the mood, the shot type, a highlight score, and the best 2–8 seconds inside it.
3. Co-write the story — you describe the film you want in your own words. The AI comes back with a beat structure grounded in what your footage actually contains — each beat names the real clips that support it — plus an honest list of anything you asked for that the footage can't deliver. You edit, reorder, delete and reweight the beats. The structure is yours.
4. Cut — the approved beats become a full timeline, shot by shot, each with the editor's reasoning attached.
5. Keep editing — trim, reorder and delete in the app, or ask for changes in a sentence ("stronger opening, fewer empty landscapes"). Then export an FCP7 XML timeline that opens in DaVinci Resolve or Premiere, relinked to your original full-quality files.

The last step is the point. Every cut is a normal clip you can trim, grade and re-time. You never get handed a locked MP4.

How we built it

A local-first Python pipeline with a browser UI:

• ffprobe for metadata, PySceneDetect for shot boundaries, OpenCV for a focus score (Laplacian variance over a 4×4 grid, 80th percentile — so a sharp horizon against a smooth sky isn't mistaken for soft focus), faster-whisper for optional local transcription, librosa for beat detection, ffmpeg for rendering.
• Claude Haiku 4.5 for per-shot logging — hundreds of cheap calls — and Claude Sonnet 5 for the storyline and timeline, one large call each. Logging 314 shots costs roughly a dollar.
• The app server is Python's standard library — no FastAPI, no npm. A teammate clones the repo and runs it without a dependency fight, and there's nothing to break on a demo laptop.

Originals never leave the machine. Only 512px stills and text go to the API. That's what makes 52 GB of 4K cheap and fast to work with, and it's a privacy property we get for free.

Challenges we ran into

Models hallucinate shot IDs. The story model occasionally invents a clip that doesn't exist, or returns in/out points past the end of a shot. Everything it returns is validated before it reaches ffmpeg: unknown IDs dropped, in/out clamped to real bounds, sub-second fragments widened or discarded, then converted to absolute source timecode so the preview and the XML can't disagree.

Short-film instincts ruin long films. Our first prompt was written for a 3-minute vlog — 1.5–4 second shots. Asked for 35 minutes, it produced a slideshow of 250 stingers. Pacing now switches on target length, and the output token ceiling scales with it, or a long timeline gets truncated mid-cut.

Footage without speech is a thin brief. With no narration the model has only pictures to reason from. Recording a five-minute voice memo about the trip and dropping it in the folder measurably improves the structure.

Cross-platform details. Windows needed three fixes before a teammate could run it: C:\ paths converted to valid file:// URLs for the XML, the Windows "address in use" errno, and console encoding (the default codepage can't print our log output).

What we learned

The prompts are the product. The pipeline took a weekend; the storyline prompt is what decides whether a cut feels like an editor made it. The highest-leverage change we made all week was asking the model to name its evidence — which real clips support each beat — and to say out loud what the footage can't do. That one instruction turned a confident-sounding generator into something you can actually trust and argue with.

What's next

Proxy playback inside the timeline, thumbnails on the beat cards, draggable trim handles, FCPXML for Final Cut, CLIP embeddings to collapse the six near-identical sunset takes everyone shoots, and reverse-geocoded place names so a beat can say "Nassau" instead of "Day 3".

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

What we learned

What's next for HEJA

Built With

• api
• claude
• fcpxml
• python