Cutting

Agentic Video Editor

Comment

Inspiration

I originally just wanted a tool that suggests background music for videos. But I quickly realized: the music choice depends entirely on the edit - the pacing, the mood, which clips come first. You can't pick music for a video that hasn't been cut yet. So the scope grew. If I'm going to get music right, I need the edit first. If I need the edit, why not let AI handle the whole pipeline? That's how Cutting was born - an autonomous agent that takes raw clips and a brief and produces finished, export-ready video edits with no human steps in between.

What it does

You upload a handful of video clips (travel footage, product shots, event coverage - whatever you have). You write a brief like "30-second energetic travel ad" and hit generate. You don't confirm anything, pick clips, or tweak settings between steps. The agent does everything.

It watches every clip, analyzes the visuals and audio, decides which moments are strongest, and assembles multiple complete edited versions - each with a different creative approach. One might be fast-paced with rapid cuts, another slow and cinematic.

Then it layers on more: AI-generated voiceover (scripted by Gemini, rendered by Cloud TTS), licensed background music from Jamendo (auto-ducked under speech), and on-screen text - titles, lower thirds, captions timed to the voiceover, end cards.

You preview each version in a timeline editor, compare them, and export your pick as an MP4 with everything baked in - transitions, filters, mixed audio, burned-in text.

Autonomous Agent Decisions

Every one of these is the agent's call, not the user's:

  • Watches raw footage and identifies strong moments - Gemini 3.5 Flash receives raw video bytes, analyzes visual content, mood, energy, and quality, and pinpoints the best timestamps. (link to code)
  • Skips clips that don't fit - the agent decides "this clip doesn't match the brief" and leaves it out. It makes editorial judgments about what belongs and what doesn't. (link to code)
  • Re-analyzes when it finds gaps - the agent reviews its own analysis, notices missing detail, and re-calls the tool with a specific focus (e.g., "look more closely at seconds 5-8 for transition opportunities"). Up to 10 re-analysis calls per clip, self-directed. (link to code)
  • Chooses creative angles for each version - each version gets a different editorial philosophy: pacing, mood emphasis, clip selection, ordering. Each version is generated in its own separate reasoning call, so they develop independently. (link to code)
  • Assembles the timeline - selects which clips, in what order, trimmed to what timestamps, with transitions (crossfade, wipe, fade) placed between segments based on mood shifts. (link to code)
  • Applies visual treatments - per-segment filters (warm, cool, sepia, grayscale) and brightness correction to smooth visual jumps between adjacent segments. (link to code)
  • Controls audio volume per segment - generates volume keyframes for the original clip audio based on audio quality and content. (link to code)
  • Scripts voiceover - writes chunked speech aligned to the timeline, with per-chunk speaking rate adjusted to match the segment's energy. (link to code)
  • Picks the best music track - searches Jamendo by mood keywords derived from the analyses, then Gemini picks the best fit from the results and decides a start offset to skip intros. (link to code)
  • Auto-ducks music under speech - generates a dynamic volume envelope so music ramps down before every voiceover chunk and ramps back up after. (link to code)
  • Decides on-screen text - places title cards, lower thirds, captions (timed to speech), and end cards. Chooses position, size, color, and timing for each. (link to code)

The agent runs all of this in a deliberate order it determines: analyze everything first (concurrently), then generate edit plans independently, then enrich each version with speech -> music -> text. The ordering matters - speech runs first so music can duck under it and captions can time to it.

The system flow

Agent framework

Google ADK Agent
================
All tools powered by Gemini 3.5 Flash (Vertex AI, global endpoint)

Step 1: Concurrent Analysis                        --> Gemini 3.5 Flash
|
|  [Analyze Clip]            [Analyze Audio]
|   Segments, mood,           Audio type, volume,
|   energy, best moments      speech, quality
|   (re-callable 10x)         (re-callable with focus)
|
v  All clip analyses
|
Step 2: Independent Edit Generation (x N)          --> Gemini 3.5 Flash
|
|  [Generate Edit Plan]
|   Unique creative angle per version
|   Returns: clips, order, transitions, filters,
|   brightness, audio keyframes, skipped clips
|
v  Complete edit per version
|
Step 3: Per-Version Enrichment (sequential)
|
|  [Speech Script]  -->  [Background Music]  -->  [Text Overlays]
|   Gemini writes         Jamendo search +         Gemini decides
|   chunked voiceover     Gemini picks best +      titles, captions,
|   (rendered via         auto-duck keyframes      lower thirds,
|    Cloud TTS)                                    end cards
|
v  Finished version --> Export via ffmpeg --> MP4

Google ADK agent (link to code) with six tools (link to code). The agent decides tool order and frequency autonomously. Each tool returns strict JSON -- creative reasoning happens in Gemini, structured contracts flow between services.

Backend orchestration

FastAPI on Cloud Run (link to code). The agent runner launches via ADK's InMemoryRunner and watches the event stream. As each tool completes, its output is persisted to Firestore immediately. If the run fails halfway, analyses already stored are still usable. The frontend shows live progress as the agent works.

Audio pipeline

Speech chunks are rendered through Google Cloud TTS (link to code) with per-chunk voice and rate, uploaded to GCS. Music is searched and selected via the Jamendo API. The auto-duck algorithm (link to code) builds a volume envelope that reacts to speech:

# Automatically duck music volume under every voiceover chunk
for chunk in sorted(speech_chunks, key=lambda c: c["start_time"]):
    # Ramp down before speech starts
    keyframes.append({"time": chunk["start_time"] - duck_fade, "volume": normal_volume})
    keyframes.append({"time": chunk["start_time"], "volume": ducked_volume})
    # Hold low during speech, then ramp back up
    keyframes.append({"time": chunk["end_time"], "volume": ducked_volume})
    keyframes.append({"time": chunk["end_time"] + duck_fade, "volume": normal_volume})

This happens automatically - music dips under speech the way a professional mix would.

Export engine

The exporter (link to code) renders the final MP4 via ffmpeg: concatenate/trim segments, apply transitions, layer in filters and brightness correction, build a 3-track audio mix (original + voiceover + music with keyframe volume), and burn text overlays via drawtext. Volume keyframes are translated into piecewise linear ffmpeg expressions.

Storage

Firestore (link to code) for all structured state. GCS (link to code) for all binary assets (clips, speech WAV, exported MP4s). Async background execution - the agent runs without blocking HTTP.

Frontend

React 19 + Vite + Tailwind (link to code) with a timeline editor, real-time playback engine, and an AI edit log showing the agent's reasoning.

Data sources

  • Jamendo - licensed instrumental music (searched via API, streamed via URL)
  • Pexels - stock video clips used for demo/showcase projects

Challenges we ran into

  • ADK telemetry crash - ADK's trace_call_llm() tries to JSON-serialize the full LLM request. When that request contains raw video bytes, it crashes. We patched it by monkey-patching the telemetry functions to no-ops before starting the agent.

  • Gemini 3.5 global endpoint - Gemini 3.5 Flash returns 404 on regional endpoints (us-central1). It only works on the global endpoint via Vertex AI.

  • 3-layer audio sync - mixing original clip audio (with per-segment volume keyframes), positioned voiceover chunks, and a music track (with fade + duck keyframes) into one coherent output required careful ffmpeg filter graph construction.

  • Veo entitlement wall - we built video generation with Veo but our GCP project only offers it via provisioned throughput (not on-demand). "Available in Model Garden" doesn't mean "callable from your project."

What we learned

The agent reasons freely, but every tool returns strict JSON that the rest of the system consumes deterministically. That separation - creative reasoning in Gemini, structured output contracts between services - is what makes the whole pipeline reliable. ADK's event-streaming model made it possible to show real-time progress without polling the model.

Surprising discoveries

  • One model does everything. Video understanding, audio understanding, editorial planning, voiceover scripting, music curation, text overlay decisions - six completely different tasks, all Gemini 3.5 Flash, all producing competent results. We expected to need specialized models for different stages. We didn't.

  • The agent actually skips clips. You'd expect an AI to use everything you give it. This one genuinely decides "this clip doesn't fit the mood" and leaves it out.

  • Re-analysis works without being forced. The agent reviews its own analyses, notices gaps, and calls the tool again with a specific focus. We gave it the capability, but the decision to use it is the agent's own.

  • Each version is genuinely different. We expected variations that are mostly the same with minor reordering. Instead, different versions often use different subsets of clips, different transitions, completely different pacing philosophies.

What's next

  • Beat syncing (cut on the beat of the music track)
  • Ken Burns motion effects on still segments
  • J/L-cuts (audio from the next clip starts before the visual cut)
  • Interactive timeline editing (drag to reorder, trim handles)
  • Media generation when we get Veo/Imagen entitlement

Built With

Share this project:

Updates

Submission history