JacRedact

Agentic Video Content Redaction

Comment

Inspiration

A support engineer records a two-minute screen capture to show a bug. It only reproduces in a real customer's account, so the video contains a real name, a real email address, and a real invoice total. Sending it to the vendor leaks all three. Not sending it means the bug never gets fixed.

Everyone we described this to had their own version of it — a founder demoing with production data, a teacher recording a walkthrough of a live gradebook, a nurse capturing an EHR workflow for training. The workarounds are uniformly bad: re-record against seeded fake data, which takes an hour and frequently doesn't reproduce the bug at all, or open a video editor and keyframe black rectangles by hand.

So we set out to automate the black rectangles, and walked straight into the thing that makes this genuinely hard.

Redaction is all-or-nothing. A two-minute recording at 30 fps is 3,600 frames, and whoever receives the video can step through it one frame at a time. A single frame where the email is legible defeats the entire redaction — not partially, completely. There is no partial credit, and no way to be mostly private.

That changes what "accurate enough" has to mean. If a per-frame detector is right 99.9% of the time and its errors were independent, a 3,600-frame video comes out clean with probability 0.999³⁶⁰⁰ ≈ 0.027 — under 3%. To ship a clean video even half the time, you'd need to be better than 99.98% correct on every single frame. This is not a bar you clear by fine-tuning a detector a bit harder.

And in reality the errors aren't independent. They cluster, and they cluster somewhere very specific: the moments when content enters or leaves the screen. A model judging one frame in isolation reliably finds text that is fully rendered and legible. It reliably misses text that is:

  • half scrolled off the top of the viewport
  • covered by a dropdown or tooltip except a two-character sliver
  • partly typed — the value is on screen, just not finished yet
  • smeared by motion blur during a fast scroll

In every one of those cases, the fragment on screen is not, to a model looking only at that frame, "the customer's email address." It is still the customer's email address.

Here is the observation that reframed the entire project for us: a single frame cannot distinguish content that has left the screen from content the model failed to see. Both are simply absence. The information needed to tell those apart does not exist in the frame. It exists in the frames on either side.

That is a statement about structure, not about model quality. No amount of better single-frame vision fixes it. So we stopped trying to build a better detector and started building something that could walk a video.

What it does

JacRedact takes a screen recording and a plain-English instruction — "redact the customer's email and anything in the spreadsheet's price column" — and returns the same video with those things burned out of every frame they appear in, including the frames where they're only half visible.

The request becomes a policy you can inspect. An LLM call turns your sentence into a typed predicate tree: literal-text conditions, on-screen-region conditions ("the browser address bar"), and semantic conditions ("a person's full name"), combined with and/or/not. That tree is stored as real nodes and edges hanging off the graph root, rendered back to you in the UI, and keyed by a hash of the request — so describing the same policy again costs nothing, and a policy you wrote for one video can be applied to the next one.

The video becomes a graph. ffmpeg extracts frames; perceptual hashing (pHash for global change, dHash for local motion) collapses runs of visually identical frames, which is what makes it affordable to put every remaining frame to a vision model. What survives is linked into a chain of Frame nodes joined by Next edges, with each frame's OCR words stored on it. That chain is the road the agents later walk.

Detection reads the frames. Every kept frame goes to the model along with its indexed OCR word list, and the model answers with which words to redact — never with coordinates. Models are good at judgement and bad at pixels, so geometry comes from Textract, and adjacent picked words get merged into a single bar rather than a row of tiles over each word.

Then agents go where detection stopped. Wherever a value is present on one frame and absent from its neighbour, that boundary is exactly the ambiguity we couldn't resolve — gone, or missed? So an agent launches there and walks the frame chain into the gap, carrying only the values that went missing, asking one question per frame about all of them at once. It draws a box wherever a value is still visible in any form, drops values the model confirms are truly gone, and stops when it has nothing left to follow or reaches ground another agent already covered.

The number of agents that launch is a direct readout of how much single-frame detection missed.

Every box records which agent placed it, which frame that agent started from, and the model's one-line reason — so any redaction in the finished video can be traced back to the decision that produced it. Finally, ffmpeg burns the boxes in across each kept frame's slice of the source timeline.

How we built it

The entire product is written in Jac — the graph, the pipeline, the agents, the web layer, and the UI. Python is called only as a leaf for ffmpeg, perceptual hashing, and Textract.

The graph is the pipeline state, not a serialization of it. Video, Frame, Box, Track, Agent, and the policy nodes all hang off root, and in Jac persistence is reachability — attach a node to something reachable and it is durable, with no ORM, no migrations, and no save call. A video and every detection derived from it survives a server restart, so an expensive multi-agent run can be reopened and replayed instead of recomputed.

Policy evaluation has no evaluator. EvalPredicate is a walker whose abilities are selected by the type of node it lands on: TextLeaf matches against OCR words, RegionLeaf and SemanticLeaf ask the model, and AndOp / OrOp / NotOp combine their children's answers in exit abilities, which Jac runs post-order — so by the time an operator computes, its children have all reported. There is no kind-switch and no recursive evaluate function anywhere in the codebase. Adding a new kind of condition is one archetype plus one ability.

A cascade agent is a walker, so the walk is the algorithm. Cascade moves along Next edges (or backwards along them, for the reverse direction), and its arrival ability runs on each frame. There is no queue to drain, no visited set, and no orchestrator: the agent's position on the graph is its state, and disengage is how it stops. Boxes it creates point back at its Agent node through an AddedBy edge, which is how provenance comes for free rather than being a logging system we had to write.

byLLM makes the model a typed executor. Every model call in the project lives in one file as a signature with no body — def plan_policy(request: str) -> PolicyPlan by llm() — where the return type is the schema and sem annotations supply the prompt. There is no prompt string to drift out of sync with a parallel Zod/Pydantic definition, because there is no parallel definition.

The UI is Jac as well. The frontend is React written in Jac's client codespace, calling server functions directly with no route table, fetch layer, or generated API client in between. The component that draws each agent's walk imports its type straight from the server module that produces it, so renaming a field fails the type check on both ends at once.

Challenges we ran into

Almost every serious bug in this project was silent — the code ran, the output looked plausible, and it was wrong. For a redaction tool, that is the worst possible failure mode, because a confidently-wrong result is a video you publish believing it's safe.

The concurrency bug that redacts the wrong words. Jac's flow doesn't capture its arguments: flow work(x) compiles to a zero-argument lambda, so every name in the expression is read again on the worker thread, after the launching loop has already moved on. Fan out from a loop and every task in the batch silently receives the last iteration's arguments. Nothing raises, the result shapes are all correct, and in our case it meant a frame's boxes being computed from a different frame's words — plausible text, wrong coordinates. Consecutive frames of a screen recording look nearly identical, which is exactly what makes this invisible.

Worse, the bug hides itself: the first fan-out in a process has to spin up a thread per task, which gives each worker time to read its arguments before the launcher advances — so a cold test passes and every subsequent fan-out corrupts. We isolated it in a spike with no network calls, then moved all model concurrency to async/asyncio.gather, since a coroutine binds its arguments when it's constructed and cannot lose them this way.

Storing the wrong things as nodes made the pipeline 14× slower. Graph writes cost time proportional to the size of the whole graph, so inserting N nodes is O(N²). Storing a video's OCR output as ~9,000 Word nodes made one pass take 345 seconds; the identical data stored inline on each frame takes 24. The lesson we wrote into our own docs: nodes are for things the pipeline traverses between — frames, boxes, tracks, agents, policy. Bulk data is a field.

ffmpeg dies silently above ~3,600 chained filters. SIGBUS, empty stderr, and an exit that's easy to miss. A long export would appear to succeed and ship a completely unredacted video. We now split the export into sequential passes of 1,500 filters and explicitly check for the signal, because "the export failed" is a recoverable outcome and "here is your safe video" when it isn't is not.

Others that cost us hours: a walker's traversal is queue-driven, so a shared depth counter records the order nodes were reached rather than how they nest — it produces a tree that looks right and isn't. One request cannot delete nodes and then create new ones; the new ones are discarded with a 200 response and nothing logged. And a streaming generator's body runs after the response has begun, at which point root is no longer the caller's root and every graph query inside quietly returns empty.

Accomplishments that we're proud of

The cascade genuinely closes the leak it was built for. Detection alone leaves the half-scrolled and popup-occluded frames uncovered; the agents find them, and you can watch the coverage count climb frame by frame as they walk.

We're also proud that the system can explain itself. Every box in the output names the agent that placed it, the frame that agent launched from, and the model's reason for believing the content was still on screen. That's a property of storing the pipeline as a graph rather than a design goal we had to implement separately — and for a privacy tool, "why did you redact that, and what makes you sure you got all of it" is the question that actually matters.

And we shipped every constraint we discovered into a document in the repo, with the measurement behind it. The concurrency bug in particular is the kind of thing that would have silently corrupted someone else's project.

What we learned

Design the failure mode first. We spent our early hours on detection quality and our productive hours on the structure that makes a miss recoverable. For an all-or-nothing task, "what happens when the model is wrong" is the more important question than "how often is it right."

Put the model where judgement lives and nowhere else. Our accuracy improved most when we stopped asking the model for bounding boxes and started asking it to pick indices from an OCR list. It's excellent at "is this the customer's email" and unreliable at "what pixel does it start at" — and the OCR engine is the reverse.

Arrival-based dispatch removes a whole category of code. Both the policy evaluator and the cascade agent are things we've written before as recursive functions with explicit queues and visited sets. Expressed as walkers, that scaffolding simply isn't there, and what's left reads like a description of the algorithm.

Correlated errors cut both ways. The clustering that makes per-frame detection fail — misses bunching at transitions — is the same property that makes neighbouring frames informative. The structure of the problem contained the structure of the solution.

What's next for JacRedact

  • Human-in-the-loop correction. Let a reviewer draw, drag, or delete a box in the UI and have that correction propagate along the track through the same agents — the graph already has the identity links to make an edit follow content across frames.
  • Beyond text. Faces, ID documents, and browser chrome are the same problem with a different detector at the leaf; the policy graph and the cascade don't change.
  • Audio. Spoken PII is the obvious gap in a screen recording, and a transcript is just another chain hanging off the video with timestamps to align against.
  • Blur and pixelate modes. Solid bars are the safest default but the least presentable; some content wants to remain legible as shape without being readable.
  • A verification pass. A second, adversarial walker whose only job is to re-read the exported video and try to find anything the policy said to redact — turning "we think it's clean" into a checkable claim.

Built With

Share this project:

Updates

Submission history