Inspiration
Video editing tools are excellent at manipulating clips, but they still leave creators with the hardest part: deciding what belongs in the story.
For talking-head videos and vlogs, creators often spend hours reviewing footage, removing mistakes, finding the strongest opening, rebuilding the narrative, and deciding where context or B-roll is needed. Existing AI editors usually behave like black boxes: they produce a result, but rarely explain why a moment was removed, moved, or emphasized.
We created StoryCut to explore a different approach—an explainable, human-in-the-loop rough-cut agent that treats editing as a storytelling problem before treating it as a rendering problem.
What it does
StoryCut turns raw talking-head or vlog footage into an editable rough-cut proposal.
The workflow is:
- Upload a raw video.
- Transcribe the dialogue with timestamps.
- Analyze representative frames and spoken content.
- Use GPT-5.6 to identify mistakes, repetition, story beats, emotional moments, pacing issues, and B-roll opportunities.
- Generate structured editorial decisions such as:
- KEEP — an important idea, reaction, or story beat.
- CUT — a mistake, repetition, or low-value section.
- MOVE — a segment that would work better elsewhere.
- B-ROLL — a moment that needs visual evidence or context.
- Let the creator accept, reject, or modify every suggestion.
- Export a rough-cut video, subtitles, and an editable timeline representation.
Every recommendation includes a short explanation. StoryCut is not intended to replace the editor; it helps the editor reach a strong first cut faster while keeping creative control visible and reversible.
Editing modes
Our initial prototype focuses on two common workflows:
Talking Head Clarity
This mode removes mistakes and repeated ideas, improves logical flow, and preserves the speaker's natural delivery.
People-first Vlog
This mode prioritizes people, relationships, questions, discoveries, reactions, and emotional payoff instead of simply arranging travel footage chronologically.
How we built it
StoryCut uses a hybrid local and AI-assisted pipeline.
- FFmpeg handles media inspection, audio extraction, clip rendering, and final rough-cut assembly.
- Whisper creates a timestamped transcript locally.
- Representative frames and transcript segments are converted into structured context.
- GPT-5.6 performs the editorial reasoning and returns a validated JSON edit plan.
- A review interface presents each recommendation with its source timecode, action, reason, and confidence.
- Approved decisions are converted into subtitle files and an executable FFmpeg timeline.
We used Codex throughout development to plan the architecture, define the edit-decision schema, scaffold the application, debug media-processing commands, validate model output, and iterate on the review experience.
One important design choice was to separate editorial reasoning from media execution. GPT-5.6 proposes what should happen and why; deterministic tools validate the timecodes and render the approved result.
Challenges we faced
Turning creative judgment into structured data
Editing decisions are subjective and interconnected. A sentence that appears repetitive may still be necessary for emotional pacing or context. We needed a schema that could represent both the proposed action and the reasoning behind it.
Keeping timestamps reliable
Transcription boundaries, video frame rates, and FFmpeg timestamps do not always align perfectly. Small timing errors can create clipped words or awkward cuts, so StoryCut validates and pads edit boundaries before rendering.
Balancing automation and creative control
A fully automatic editor can make confident but undesirable decisions. We designed StoryCut around reviewable suggestions instead of irreversible automation.
Managing long media efficiently
Sending entire videos to an AI model would be slow and expensive. StoryCut performs transcription and media preprocessing locally, then sends compact, relevant context for higher-level reasoning.
What we learned
We learned that the most valuable role for AI in editing is not adding more effects. It is helping creators reason about structure, intention, and audience attention.
We also learned that explainability improves the product experience. Showing why a cut was proposed makes it easier for creators to trust, reject, or improve the result.
Finally, combining probabilistic model reasoning with deterministic media tools produces a more reliable workflow than asking one system to do everything.
What's next
Next, we plan to add:
- Multi-camera interview support.
- Personal editing-style profiles.
- Timeline export for professional editing software.
- Better visual analysis for reactions and scene changes.
- Collaborative review and comments.
- Evaluation tools that compare AI recommendations with human editor decisions.
Our goal is to make StoryCut a practical editorial partner: fast enough to remove repetitive work, transparent enough to earn trust, and flexible enough to preserve the creator's voice.
Log in or sign up for Devpost to join the conversation.