Frameflow

An agent that triages a film for 270° immersive conversion, recovers the side walls from the footage itself, and never lies about what it invented.

Comment

Inspiration

I went looking into how films actually get converted to the 270° three-wall format, and the numbers stopped me. It takes about two months per title. Two or three weeks just moving assets around, four or more weeks of CG, then two weeks of QC with the studio. And after all that, you don't even get the whole film — Bohemian Rhapsody got 43 minutes out of 134.

The work itself is artists digging through alternate takes and B-roll and rotoscoping bits of them into the side panels. Painstaking, and expensive enough that the format mostly doesn't get used.

But a lot of that time isn't spent on the art. It's spent figuring out which shots are even possible in the first place. If the camera panned, it already filmed the side walls — the crop just threw them away. If it was locked off on someone's face, that footage never existed and never will, no matter how many hours you throw at it.

That's not an art problem. That's geometry, and geometry is cheap. Figuring it out first is the difference between converting 43 minutes and converting the film.

What it does

You give it a film. It tells you which shots can be widened using footage that already exists, converts those, refuses the rest, and records every decision — including the refusals — somewhere you can query later.

There are three agents and the split between them matters:

scout decides what's worth doing, and it can't render. That's deliberate. It runs the same motion classifier and geometry probe a real render uses, just on a window of each shot instead of all of it. Seconds per shot instead of hours per film.

conversion does the work and doesn't get an opinion on whether it was worth doing. Renders the three projector feeds, settles the walls, measures what came out.

archivist keeps the records and never touches a pixel.

The thing I care most about: every pixel in a side wall is either photographed or invented, and Frameflow will not let those blur together. There are six provenance rungs, from primary down through recovered and donated to generated. mean_real_wing is how much of the wall is genuinely photography. If a model repaints part of it, that number drops by exactly the amount it repainted, and the per-pixel map on disk gets rewritten to match. You can't quietly launder invented pixels into the good column.

And when the answer is no, it says no. A **LOCKED** shot means the camera never filmed anything out there. I'd rather it tell you that than invent something and call it a conversion.

How we built it

Gemini 2.5 Flash on Vertex AI, with Google Cloud Agent Builder (ADK) running a supervisor over the three sub-agents. It's on Cloud Run with --no-cpu-throttling, which I didn't set at first and immediately regretted — background renders were taking 5.5 minutes that should have taken under two, because the CPU was being throttled the moment the request finished.

ClickHouse isn't bolted on here. The question a studio actually asks isn't how did this one shot do. It's "across everything we own, what converts without us inventing anything?" That's a query. And it only works if the refusals are rows too — a table of successes can tell you what you converted, but never what you could have.

**SELECT** source,
    count()                                        AS shots,
    countIf(state IN ('**FULL**','**NARROW**','**BORROWED**')) AS earned,
    round(avg(photographic), 4)                    AS mean_real_wing
**FROM** frameflow_shots
**GROUP** BY source
**ORDER** BY earned **DESC**

One row per shot, carrying its motion class, backend, geometry score, verdict, coverage, every provenance rung. Reads go through the official mcp-clickhouse MCP server so the agent writes its own queries instead of picking from a handful I thought of in advance.

The geometry underneath is OpenCV and NumPy — homography propagation, a leave-one-out probe that hides a frame and scores how well the others predict it, and a gate that turns that score into FULL, NARROW, or OFF.

Challenges we ran into

The black lines. For a while the recovered walls had thin dark lines running down them and I assumed that was just what this approach looked like — a limitation I'd have to document and live with. It wasn't. Donor frames were being warped with bilinear interpolation against a black border, while the masks saying which pixels were valid got warped nearest-neighbour. So the edge column of every donor came out at about half brightness, and got labelled as real photography. I reproduced it on a flat grey image: source 180, output 90, mask says valid. Went from 2.67% of wall columns affected to 0.00%.

That one stung, because I'd already half-convinced myself it was unfixable.

A metric that punished me for doing better. Effective coverage kept dropping when I raised the resolution. Turned out the detail weighting used a fixed 9-pixel window, so the same wall at higher resolution just looked less detailed to it. It was reading 24.57% where the truth was 52.54%.

Four different bugs wearing the same costume. Getting the ledger connected took four rounds, and every single failure said ledger not connected. First the package wasn't installed. Then mcp 2.x had removed mcp.shared.session. Then uvx was downloading the server at call time and blowing through ADK's timeout. Then the timeout was 5 seconds and a ClickHouse service that's scaled to zero takes longer than that to wake up. Same message every time.

Tests that were lying. Each suite collected results in a **FAIL** list and also ran standalone, which was fine — except pytest never looked at that list. So a test could record five failed checks, return normally, and get reported as passing. I found this by accident. Added a conftest.py hook to fail any test whose module recorded a failure, and it immediately caught two more assertions that had been quietly wrong for who knows how long.

Accomplishments that we're proud of

Triage guesses low, and I can prove it. On a 27-second handheld pan it took 91 seconds and predicted 50.7% coverage. The full render took two hours and came back with 52.54%. That's the direction you want to be wrong in. A window has fewer donor frames than the whole shot, so a real render always recovers more, never less. Guessing low costs you a shot you could have had. Guessing high costs you the artist-month you already committed.

The walls are actually real. Three rooms, three different days, CPU only, no model involved anywhere:

clip genuinely photographed wall
café 98.7%
apartment walk up to 99.3%
gym pan, 1024px, whole take 90.4%

The scout genuinely cannot render. This sounds like a small thing and isn't. Give one agent all the tools and ask it whether a shot is worth converting, and it'll just render it to find out — which is precisely the cost the whole thing exists to avoid. Taking the tool away is what makes the answer trustworthy. There's a test that checks it's still gone.

618 checks across 10 suites, all passing.

What we learned

Look at the output. My metrics were telling me things were fine while the walls were visibly shimmering and had lines down them. I pulled some frames, looked at them, and found both problems in about a minute. After that I stopped trusting numbers alone for anything visual.

Bugs live in between things, not inside them. Every unit test stayed green through every bug I actually shipped. The one where the film came out a quarter too slow survived because the test fixture was 24fps and happened to agree with three separate hardcoded defaults. Writing an end-to-end test with a deliberately 30fps fixture is what finally caught it.

Saying no is a feature. About three quarters of the shots in a fast-cut trailer come back **LOCKED**. A static camera has no periphery — that's just true. A tool that admits it is more useful than one that always hands you something, because the second kind can't be trusted anywhere.

What's next for Frameflow

The walls still repeat signage that's visible in the centre of frame, and I know why: one homography can't put off-plane objects at the right offset. The fix is GaussianBackend, which reconstructs the scene in 3D and renders a wider frustum. It needs CUDA, I've verified the path works on a T4, but I haven't run it across a full title yet.

After that I want the **DONATED** and **RETRIEVED** rungs to actually fire on real footage — so a locked-off close-up can borrow from the master wide that filmed the same wall an hour earlier. Right now cross-setup matching verifies about 1 in 400 candidate pairs. That number's honest and it's not good enough yet.

And the real test: running it across a whole catalogue instead of one film. That's the question the ledger was built for, and it's the one that decides whether any of this is worth a studio's time.

Built With

Share this project:

Updates

Submission history