• 

  Vhideo stream highlighter test (works!)

FrameMill — Devpost Project Story and Reproduction Guide

About the project

Inspiration

A long gaming stream can contain hours of footage but only a few moments worth publishing. Finding those moments, checking what actually happened, arranging them into a coherent edit, and moving the result into an editor is repetitive work that usually starts after the stream ends.

FrameMill explores a different workflow: the video arrives once, deterministic game processing records candidate Activities, and a Gemini-powered editorial agent turns verified evidence into editable productions. The creator remains in control of the final edit rather than receiving an unexplained black-box video.

What it does

FrameMill is a cloud production desk for turning gameplay video into three useful starting points:

• a concise vertical-ready Short;
• a coherent full-game edit;
• a chronological kill-highlight reel containing verified enemy downs and eliminations.

The upload is stored privately in Google Cloud Storage and registered with its exact immutable object generation. The control plane creates one production for each saved template, persists the session and jobs in Firestore, and starts a private Cloud Run Job. The production worker uses Google ADK and Vertex AI. Planning and editing use gemini-3.7-flash; bounded media interpretation uses gemini-3.5-flash-lite. Activity records are treated as fallible notes: Gemini must verify the corresponding source frames and discard unsupported events.

The result is a provenance-linked framemill.timeline.v1 edit that can be projected into the included OpenReel-based editor instead of being locked into an opaque rendered output.

How we built it

FrameMill is divided into explicit ownership boundaries:

1. A deterministic Go processor produces factual, timestamped gameplay Activities.
2. Google ADK agents expose typed tools for retrieving evidence, planning an edit, creating a Timeline, and tracking durable work.
3. The authenticated Go control plane runs on Cloud Run and owns users, templates, sessions, productions, jobs, and private artifact access.
4. Firestore stores durable coordination state. Cloud Storage stores source media and generated artifacts with generation-qualified references.
5. A private Cloud Run Job executes the Gemini production worker.
6. The React client uploads directly to a narrowly authorized resumable Cloud Storage session, then registers the exact completed object generation.

The three default templates are installed idempotently for the demo user. A successful upload automatically dispatches all three, with stable idempotency keys so a retry cannot accidentally create duplicate work.

Challenges

The hardest challenge was keeping AI decisions grounded. Processor events can be noisy or incorrect, so FrameMill does not treat an Activity label as proof. The agent receives it as a candidate note and requests bounded visual evidence before including it in an edit.

Cloud security was another important constraint. The media bucket is private, public access prevention is enforced, the browser receives only a short-lived resumable upload capability, and application state stores the exact object generation. Runtime identities are separated for builds, the control plane, and production jobs.

Finally, the application has to remain usable when a secondary panel fails. For example, job-history loading is isolated from upload capabilities so a transient Firestore query problem cannot disable source uploads.

What we learned

• Agentic editing works best when deterministic perception and model judgment have separate contracts.
• Evidence provenance is as important as the generated Timeline.
• Idempotency and durable job state are essential once an agent launches cloud work asynchronously.
• A useful AI video product should hand creators an editable artifact, not only a final render.

Reproduce the hosted demo

The current demonstration is deployed at:

https://framemill-app-457172817004.us-central1.run.app/

Use the evaluator account supplied in the private Devpost judging instructions; credentials are intentionally not committed to the public repository.

1. Sign in and choose New session.
2. Select Existing file and choose a small MP4. A 30–60 second gameplay

Built With

• codex
• go
• vue