HEJA Vlog.ai

Main page

Inspiration

In April I came home from a week in Miami and the Bahamas with 250 drone clips — 52 GB, 63.6 minutes of 4K. Six months later I had edited exactly none of them.

That's not a motivation problem. The hard part of editing isn't cutting; it's watching. Before you can make a single decision you have to sit through an hour of footage, remember which take was the sharp one, and hold a story in your head. The cutting is the easy part.

So we went looking for a tool, and found an entire market pointed the other way.

Every AI video tool we tested — OpusClip, Descript, CapCut's AutoCut, Veed, Eddie — solves the same problem: you have one long video, and you want short clips out of it. Podcast to Reels. Webinar to TikToks. A two-hour stream to twelve hooks. The whole category is a slicing machine.

Nobody is solving the inverse: you have two hundred short clips, and you want one coherent film. The few tools that touch it dump everything into a beat-synced highlight reel — pretty, rhythmic, no story, and locked as an MP4 you can't change.

Then we started asking people whether that gap was ours alone. It isn't:

Travel vloggers come home with hundreds of clips per trip and a backlog that grows faster than they edit. Most never publish the trip at all.
Wedding and event videographers shoot six to ten hours of fragments across a single day — getting ready, first look, speeches, dances — and then spend days just logging before the edit starts. They're paid for taste, and they spend most of their hours on retrieval.
Real estate and small brand teams shoot dozens of short takes per property or product and need a consistent 60-second cut every time, without hiring an editor for each one.
Parents with four years of phone footage have the most emotionally valuable archive of anyone we spoke to, and the least chance of ever doing anything with it.

Different users, same shape: many small pieces in, one story out. That's the job nothing on the market does.

The other thing we heard, from the professionals especially: they don't want an auto-editor. They want the watching done, not the deciding. The moment a tool hands back something they can't open in Premiere, it's a toy.

So the gap was specific: nothing watches your footage, agrees on a story with you, and then hands the decisions back.

What it does

HEJA is an assistant editor, not an auto-editor. It covers the whole loop — before the shoot, after the shoot, and into your real editing software.

0 · Plan the shoot (before you travel). Tell it where you're going, with whom, for how many days, and the feeling you want. It returns a beat structure and a day-by-day shot list: what's in frame, how long to hold, which beat each shot serves, and why the edit will need it — including the connective shots everyone forgets (hands, feet walking, doors, signs, food arriving, someone looking back at camera). It flags the handful of shots the film can't be cut without, and prints a phone checklist you tick off while shooting. Shoot to the plan and the edit comes together dramatically faster, because the structure and the footage were designed for each other.

1 · Import and log. Point it at a folder. It reads capture time, GPS, duration and orientation, groups clips into days, splits long takes into shots, and scores each shot for focus and exposure so soft and blown-out takes are flagged up front. Then the AI watches every shot — keyframes plus any locally transcribed speech — and writes a log entry: what happens, who's in it, the mood, the shot type, a highlight score, and the best 2–8 seconds inside it.

2 · Protect what can't be cut. You name who the film is about ("me and my boyfriend James"). From then on the AI tags every shot for a clear face, the subject's presence, two people in frame together, and scenery value. Those become must-keep shots: the model is told it has to place them, and — because models forget — after the cut is generated the program checks and re-inserts anything that went missing, into the beat where it belongs, labelled so you can see it happened. You can also flag or exclude shots by hand, so automatic detection and your own judgement stack.

3 · Co-write the story. You describe the film you want in your own words. The AI returns a beat structure grounded in what your footage actually contains — each beat names the real clips that support it — plus an honest list of anything you asked for that the footage can't deliver ("no clean night sky; the one night clip is rainy and very noisy"). Edit, reorder, delete and reweight the beats. The structure is yours.

4 · Cut and refine. The approved beats become a full timeline, shot by shot, each cut carrying the editor's reasoning. Trim, reorder and delete directly, or ask for changes in a sentence: "stronger opening, fewer empty landscapes." Every save is versioned, so you can compare two directions and restore either.

5 · Render and export. A preview MP4 with captions, cuts snapped to the music's beat, and the music ducked under anyone speaking. Then an FCP7 XML timeline that opens in DaVinci Resolve or Premiere, relinked to your original full-quality files: picture on V1, original sound on A1 (muted where the AI chose not to keep it — just enable it), music on A2, with beat names and the AI's reasoning as markers.

That last step is the point. Every cut is a normal clip you can trim, grade and re-time. You never get handed a locked MP4.

The app runs in Chinese and English — interface, AI output, captions and beat names all switch together — with a settings page for models, frame sampling, scene-detection sensitivity, quality thresholds, concurrency and paths, and a live token estimate so you can see what a run will cost before you start it.

How we built it

A local-first Python pipeline with a browser UI.

ffprobe for metadata, PySceneDetect for shot boundaries, OpenCV for the focus score (Laplacian variance over a 4×4 grid, 80th percentile — so a sharp horizon against a smooth sky isn't mistaken for soft focus), faster-whisper for optional local transcription, librosa for beat detection, ffmpeg for rendering.
Claude Haiku 4.5 for per-shot logging — hundreds of cheap calls — and Claude Sonnet 5 for the shot list, the storyline, the timeline and the revisions, one large call each. Logging 314 shots costs roughly a dollar.
The app server is Python's standard library — no FastAPI, no npm. A teammate clones the repo and runs it without a dependency fight, and there's nothing to break on a demo laptop.
An offline mock mode replaces every AI call with deterministic rules, so the entire pipeline and UI can be developed and demoed with no API key and no cost.
A logging system writes every step to a rolling file and to an in-app log viewer with filtering and full stack traces, which is how we debugged four-hour runs on a laptop instead of guessing.

Originals never leave the machine. Only 512px stills and text go to the API. That's what makes 52 GB of 4K cheap and fast to work with, and the privacy property comes free with it.

Challenges we ran into

Models hallucinate shot IDs. The story model occasionally invents a clip that doesn't exist, or returns in/out points past the end of a shot. Everything it returns is validated before it reaches ffmpeg: unknown IDs dropped, in/out clamped to real bounds, sub-second fragments widened or discarded, then converted to absolute source timecode so the preview and the XML can't disagree.

"Don't cut this" is not enforceable with a prompt. Our first protection attempt just told the model which shots mattered. On a short cut it complied; on a 35-minute one it quietly dropped faces anyway. The fix was to stop trusting it: compute the must-keep set in code, pass it in, and then verify the output and repair it. Prompts express intent; only code enforces it.

Short-film instincts ruin long films. Our first storyline prompt was written for a 3-minute vlog — 1.5 to 4 second shots. Asked for 35 minutes, it produced a slideshow of 250 stingers. Pacing now switches on target length, and the output token ceiling scales with it, or a long timeline gets truncated mid-cut.

Footage without speech is a thin brief. With no narration the model reasons from pictures alone. Recording a five-minute voice memo about the trip and dropping it in the folder measurably improves the structure — which is also what pushed us to build the pre-shoot planner.

Cross-platform reality. Windows needed several fixes before a teammate could run it: C:\ paths converted to valid file:// URLs for the XML, the Windows "address in use" errno, console encoding, and font paths for burned-in captions. We also found and fixed a silent data-loss bug where editing a shot's in-point and saving deleted that shot from the timeline.

Accomplishments that we're proud of

The honesty feature. When the AI proposes a structure, it also says what your footage can't support. Watching it tell you "there's no clean night sky in here" is the moment people stop treating it as a slot machine and start treating it as a collaborator.

Protection that actually holds. You can watch it work: delete a must-keep shot from the timeline, regenerate, and it comes back in the right beat with a note explaining why.

It exports a timeline, not a video. Everything we built stays compatible with how professionals already work.

Two people, one week, and it runs on a clean laptop in five minutes — on macOS and Windows, with no API key needed to try it.

What we learned

The prompts are the product. The pipeline took a weekend; the storyline prompt is what decides whether a cut feels like an editor made it. The highest-leverage change we made all week was asking the model to name its evidence — which real clips support each beat — and to say out loud what the footage can't do. That one instruction turned a confident-sounding generator into something you can argue with.

The second lesson was about trust boundaries: anything the user would notice if it disappeared cannot live in a prompt. It has to be computed, passed in, verified and repaired in code. The AI proposes; the program guarantees.

And from talking to people: professionals don't want their taste automated. They want their logging automated. The tools that will win are the ones that give the last decision back.

What's next for HEJA

Proxy playback inside the timeline, thumbnails on the beat cards, draggable trim handles, and FCPXML for Final Cut.

Then the things our interviews pointed at: face clustering so you name someone once and the whole archive knows them; CLIP embeddings to collapse the six near-identical sunset takes everyone shoots; reverse-geocoded place names so a beat says "Nassau" instead of "Day 3"; and multi-day event mode for wedding videographers, where the beat structure is a known shape and the footage has to be fitted into it.

Longer term, the shot list and the editor close a loop worth owning: plan the shoot, shoot the plan, and cut against the structure you already agreed on.