PAVA Powerful Agentic Video Editor

An agentic video editor on a real multitrack timeline. It cuts, arranges, builds motion graphics, generate license-free images, video, original music - and every edit is still yours to change

Comment

PAVA — Powerful Agentic Video Editor

Category: Taskmaster

Inspiration

I used to edit video for money while I was still at school. Freelance, short-form, one client at a time.

Editing a 1–5 minute video the traditional way takes hours. Removing every "uhm", finding which take is the good one, arranging clips on a timeline, hunting for an asset that only exists in your head, adding captions. It is a lot of effort and time, and almost none of it is creative work.

Currently most AI video tools do not solve this. They hand back a finished clip. Once it is done, you cannot edit it — if one second is wrong, you re-prompt from zero and hope.

PAVA was built on one thesis: AI should do the heavy, slow, repetitive part, and the human should still be free to change anything afterwards. The agent can do everything a human editor can do, and everything it does stays editable.

What it does

PAVA is a browser video editor where the multi-step chores run themselves.

Drop a file and a background pipeline runs on its own. The upload goes straight to Cloud Storage as a resumable session, so a dropped connection continues instead of starting over. A Cloud Task is queued, and a separate Cloud Run worker wakes up: ffmpeg normalises the file, writes a scrubbable proxy, extracts audio, cuts a filmstrip, and marks the silences. Gemini 3.5 Flash transcribes the words, Cloud Speech-to-Text measures their timings, and the two are merged. Gemini then watches the proxy and returns a timed visual index — shot by shot, what is on screen and whether it is usable. Music gets a beat grid. The browser holds a Firestore listener, so the media bin fills itself in. Nobody clicks anything after the drop.

Then you type a sentence, and the agent has 34 tools. "Cut the filler words and tighten this to ninety seconds" becomes a chain: read the transcript, find the fillers, cut them, close the gaps, look at the result, trim again. Each tool sees what the previous tool wrote. It can also search every line the project ever transcribed, cut on a beat, duck music under speech, grade the whole film in a word, build motion graphics, and read its own cut back shot by shot to check whether a take belongs.

If the material does not exist, it makes it. Music from Lyria 3, images from Gemini 3 Pro Image, video from Veo 3.1 Fast. Each one lands in the bin as a real file and goes through the same ingest as an upload, so a generated track arrives already knowing its own tempo.

Everything the agent does is an operation — the same operation your mouse emits. A whole turn collapses into one undo entry: "Undo: removed 14 filler words." One press and it never happened. That is the product in one sentence: the agent does an hour of work, and you keep all the control you would have had doing it yourself.

One more press exports it. Remotion renders on the server from the same React composition that drew the preview, so what you watched is what you download.

How we built it

A Vite SPA on Firebase Hosting, and two Cloud Run services in us-central1:

Firebase Hosting (Vite SPA)
   │  Firebase ID token on every call
   ├──────────────► Cloud Run: api          ADK agent loop, SSE, signed URLs
   │                   │
   │                Cloud Tasks
   │                   ├── ingest ──► Cloud Run: worker   ffmpeg + Gemini
   │                   └── render ──►                     Remotion
   │                                        │
   ├──── direct resumable upload ────────► Cloud Storage
   └──── realtime listener on media ─────► Firestore

Splitting the API from the worker was necessary, not tidy. Cloud Run routes by concurrency, so an instance busy with a render still looks free, and a chat turn sent to it crawls. The worker image also carries Chrome and ffmpeg, which would slow the cold start of every chat request.

The agent runs on Google ADK for TypeScript with gemini-3.5-flash, reached through Vertex AI as the service's own identity. No API key exists anywhere in the deployment.

State lives in exactly one place. A turn materialises a working copy of the timeline, and every tool reads and writes that same copy — so tool eleven sees what tool ten did. Multi-step instructions depend on this completely: without it, the second tool reasons about a world that no longer exists.

The model never sees the raw document. It gets a projection — clip IDs, time ranges, transcript references. Fewer tokens, a smaller surface to hallucinate against, and the document can change without touching a single prompt.

One function writes the timeline. Agent tools and mouse gestures both go through it, and it returns the inverse of every change. That is why an entire agent turn undoes in one press.

Every tool is scoped to one user by construction. Storage handles are built for an owner and a project, and no method accepts a user ID — so another user's file is not a permission error, it is unnameable. Firestore rules refuse every write from the browser, the worker is private and accepts only calls carrying a signed Cloud Tasks token, and the build identity and runtime identity are separate service accounts.

1,300 tests, including property-based tests that fire random operation sequences at the timeline, real headless-Chrome renders compared pixel by pixel, and emulator tests proving one user's project is unreachable from another user's session.

Challenges we ran into

The transcript timings were not measured at all. Captions drifted, so we checked. On 32.0 seconds of speech, the model reported 21.0 seconds — every word on a round 100 ms grid, no gaps anywhere. It was not a bad measurement, it was a plausible guess written at a constant speaking rate. The error scales rather than shifts, so a caption slightly early at the top of a clip is ten seconds early at the bottom:

$$\varepsilon(t) \approx \left(1 - \tfrac{21.0}{32.0}\right)t \approx 0.34\,t$$

The fix was to stop asking one service for two things. Gemini says WHAT was said, keeping every filler and stammer. Cloud Speech-to-Text says WHEN, because it measures perfectly but deletes every filler. The two word lists are aligned over text, never over time. The key insight: the gaps are the fillers. If the recogniser heard one word end at 4.64 s and the next begin at 6.00 s, something took 1.36 seconds, and the model is what tells us it was "um".

Work behind a response is work that stops. Cloud Run gives a container CPU only while it is answering a request. A worker that accepts a job and then does the work behind the reply loses its CPU mid-task and leaves the job stuck forever, with nothing in the logs to explain it. Accepting a job and performing it had to become two separate operations, which is now what lets ingest and render run for minutes with nobody waiting on them.

Making generated design not look generated. Our first motion graphics were technically correct and completely forgettable. The instinct is to give the model more controls, and that made it worse — every extra field is one more thing to average, and the average of any creative space is what everyone has already seen. What worked was the opposite: fewer options, stronger ones. Seven named styles, each defined as much by what it refuses to do as by what it does. A name is a judgement a model can make well; a number is one it invents.

Accomplishments that we're proud of

  • A real editing agent, not a clip generator. 34 tools, chained inside a single turn, and every single thing it does can be dragged, trimmed or undone afterwards.
  • Four Google generative models in one product — Gemini 3.5 Flash, Gemini 3 Pro Image, Lyria 3 and Veo 3.1 — with generated media re-entering the same ingest pipeline as an upload, so a generated track arrives already knowing its own tempo.
  • A pipeline that runs with nobody watching it. Upload, normalise, transcribe, measure, analyse, index — all of it triggered by a single file drop and finished in the background.
  • Deployed and live on Google Cloud, behind Google sign-in, with real files in Cloud Storage and real renders coming out of a real worker.

What we learned

Measure instead of assuming. We had written off generated video as too slow and too expensive to belong in an interactive editor. When we finally measured it, one shot cost 48 seconds and 32 cents — well inside what a loop can afford. The assumption had been blocking a feature that turned out to be ready to ship.

Split WHAT from WHEN. Our best architectural move was noticing that one question was really two, and that different services are excellent at different halves. Gemini writes better words and cannot measure. The recogniser measures perfectly and throws the words away. Neither could do the job alone.

What's next for PAVA — Powerful Agentic Video Editor

  • Background removal for video, so the agent can lift a subject out of a shot.
  • Plan mode, so you can see what the agent intends to do before a long run starts.
  • Richer motion graphics, with more range and more distinctive design.

Built with Google ADK for TypeScript, Gemini 3.5 Flash, Gemini 3 Pro Image, Lyria 3, Veo 3.1, Cloud Speech-to-Text, Cloud Run, Cloud Tasks, Firestore, Cloud Storage, Firebase Auth and Firebase Hosting.

Built With

Share this project:

Updates

Submission history