EditDiff

Regression testing and release QA for video—verify revisions, catch accidental changes, and block bad exports before they ship.

Comment

Inspiration

Video production has a strange gap.

A creator sends revision notes. An editor makes the changes and exports a new version. Then someone still has to manually scrub through both videos asking:

  • Did every requested revision actually happen?
  • Did something else change by accident?
  • Where exactly did the change happen?
  • Is this export actually ready to publish?

Software teams have regression tests before they ship code. Video teams usually do not.

That became the idea behind EditDiff:

Regression testing and release QA for video production.

We wanted to build something that does not simply look at a video and make AI-generated claims. Every conclusion should be backed by measurable evidence.

Our core principle became:

Evidence before assertion.

What it does

EditDiff has two main product directions.

Compare Versions

This is the evidence workspace for understanding exactly what changed between two video exports.

It has two modes.

Verify Revisions

Upload:

  • V1
  • V2
  • revision notes

EditDiff checks every requested revision independently and returns:

PASS · FAIL · REVIEW

Each result includes timestamped evidence, synchronized before/after playback, measured signals, and an explanation of how the conclusion was reached.

REVIEW is intentional. If the evidence is not strong enough to prove a requested change, EditDiff does not turn uncertainty into a fake PASS.

Discover Changes

Sometimes there are no revision notes. You simply have two exports and want to know what changed.

EditDiff aligns both timelines and generates a timestamped Change Ledger containing detected:

VISUAL · TIMING · AUDIO · TEXT · REVIEW

This lets EditDiff answer both:

Did the requested edit happen?

and:

What else changed?

Release Gate

Release Gate combines those capabilities into a final pre-publish QA workflow.

Upload:

  • baseline / pre-final export
  • release candidate / final export
  • revision notes

EditDiff then checks:

  1. Requested revisions Were the requested edits actually completed?

  2. Unexpected changes Did something else change that was not accounted for by an approved revision?

  3. Technical QA Are there measurable regressions such as audio-track loss, sustained audio dropout, suspicious black-frame changes, resolution differences, or unexplained duration changes?

The final result is one of:

READY TO PUBLISH

All requested revisions passed, discovered changes were accounted for, and applicable technical checks passed.

NEEDS REVIEW

There is no definite blocking failure, but EditDiff found uncertainty, an unexpected change, or another issue requiring a human decision.

BLOCKED

A requested revision definitely failed or a high-confidence blocking technical failure was detected.

An unexpected change alone does not automatically mean the video is bad. EditDiff can prove that something changed, but it should not pretend to know whether the creator intended it.

That distinction is central to the product.

The regression-testing moment

Our Release Gate demo illustrates the problem clearly.

A requested audio mute is completed successfully:

PASS

EditDiff also detects the corresponding audio difference and correctly marks it:

ACCOUNTED FOR

But elsewhere in the same export, a logo disappears even though no revision requested that change.

EditDiff marks it:

UNEXPECTED

The requested edit was completed, but the entire export is not automatically approved.

The final decision becomes:

NEEDS REVIEW

That is the idea behind EditDiff in one example:

Passing the requested edit does not mean nothing else broke.

How we built it

EditDiff uses a hybrid deterministic + semantic architecture.

The frontend is built with Next.js, React, and TypeScript.

The backend uses Python and FastAPI, with FFmpeg, ffprobe, OpenCV, and NumPy for media analysis.

Timeline alignment

A simple timestamp-to-timestamp comparison breaks as soon as an editor inserts or removes footage.

EditDiff samples compact visual descriptors from both videos and performs bounded sequence alignment.

This lets the system distinguish:

  • matched regions
  • replacements
  • insertions
  • deletions
  • timing shifts

and realign the videos instead of producing a flood of false differences after an edit.

Visual analysis

Whole-frame averages can easily miss small but important changes such as:

  • titles
  • logos
  • lower thirds
  • crop adjustments
  • color changes

We therefore combine multiple deterministic signals:

  • global luminance difference
  • changed-pixel ratio
  • localized tile differences
  • color distance
  • edge differences
  • ORB feature matching
  • multi-frame confirmation

This makes it possible to detect both large shot changes and small localized regressions.

Audio analysis

EditDiff extracts audio and measures local RMS/envelope behavior to identify changes such as:

  • localized muting
  • major energy shifts
  • sustained audio dropout
  • complete audio-track loss

For revision verification, a requested mute is only considered a PASS when the deterministic evidence supports it.

Timing changes

Insertions and deletions are verified using timeline alignment and matching visual flanks around a changed region.

This prevents an edit early in the timeline from making every later timestamp look incorrect.

Visible text

Google Gemini is used only as an optional semantic interpretation layer.

The deterministic system must first identify and support a real visual difference.

Only then can Gemini inspect a bounded set of aligned evidence frames and help classify readable text changes.

Gemini is not allowed to invent where a change happened.

If Gemini is unavailable, times out, returns malformed output, or cannot confidently interpret the evidence, EditDiff keeps the deterministic result instead of fabricating certainty.

Our architecture can be summarized as:

AI interprets evidence. It does not invent the evidence.

Challenges we faced

Timeline drift

Comparing identical timestamps fails immediately after an insertion or deletion.

We solved this with bounded sequence alignment and independent flank verification.

Small visual edits

Early versions were good at detecting full-shot replacements but could miss small logos, titles, crops, and color changes.

We rebuilt visual detection around localized and structural signals rather than relying on one global frame-difference score.

Compression noise

Lowering thresholds helps find smaller edits, but can also make re-encoded copies look different.

We added regression cases covering different CRF values, bitrates, scaling, and re-encoding paths so compression alone would not become a meaningful change.

Requested vs. unexpected changes

Release Gate introduced a new problem: a discovered difference should not be labeled unexpected if it is simply the evidence of a requested revision that passed.

We built conservative correlation logic between revision evidence and discovered change windows.

A change is only treated as accounted for when the relationship is strong enough. Ambiguous relationships remain REVIEW instead of being silently hidden.

Technical QA without false confidence

Video production contains intentional silence, still frames, black slates, resolution changes, and timing changes.

We deliberately kept the first technical QA layer conservative rather than shipping a long checklist full of unreliable alarms.

What we learned

The biggest lesson was that useful AI tooling does not always mean giving the model more control.

EditDiff became more reliable when we separated:

detection

from:

interpretation

Deterministic systems handle measurable facts.

AI is used where semantic understanding is genuinely useful.

We also learned that uncertainty should be part of the product rather than treated as an error.

A trustworthy system should be able to say:

“I found evidence, but I cannot prove the claim.”

That is why REVIEW is a first-class result throughout EditDiff.

Finally, Release Gate changed how we thought about the product.

We started by asking:

Did the requested revision happen?

Then:

What else changed?

The larger question became:

Should this export ship?

That turned EditDiff from a video-difference tool into a release-quality workflow.

Current result

EditDiff can now perform three connected jobs:

Verify

V1 + V2 + revision notes → PASS / FAIL / REVIEW

Discover

PRE-FINAL + FINAL → timestamped Change Ledger

Release Gate

Baseline + candidate + revision notes → requested-revision QA + unexpected-change detection + technical QA + release decision

The goal is simple:

Stop manually hunting through exports. Know what changed, prove what landed, and catch regressions before they ship.

Software has regression tests.

Video production should too.

Built With

Share this project:

Updates

Submission history