Provena

Stop paying twice for AI training video. We catch duplicate, edited and reused footage in every delivery, show the proof, and let AI decide what to pay within fair, fixed rules.

Comment

Inspiration

AI companies buy huge amounts of video to train their models, and they usually pay by the hour of footage. But almost nobody checks what they're actually getting. We surveyed the 300 most-downloaded video datasets on Hugging Face and fingerprinted five of the biggest, OpenVid-1M, LLaVA-Video-178K, Vript, ShareGPT4Video and VideoChat3: 1.63 million clips and about 11,500 hours of footage. In OpenVid-1M alone, where we compared every clip against every other, about 8% of the footage (155.6 of 2,050 hours) is duplicate or near-duplicate: the same clip listed twice, recoloured copies, trimmed and shifted cuts. If that happens in free research datasets, it happens in paid deliveries too, where every duplicate hour is money paid for nothing. And the people hurt most by sloppy checks are often honest contributors, who get paid per hour and can lose pay to a bad automated call.

What it does

Provena Gate checks a video delivery before the invoice is paid. A vendor delivers a batch of clips. Provena Gate finds four kinds of footage that shouldn't be paid for:

  • the same clip twice in one delivery
  • edited copies of footage the buyer already owns
  • footage already sold to another buyer (only if both buyers opt in; neither ever sees the other's footage)
  • clips taken from public AI training datasets (checked against 1.63 million public clips)

It catches copies even after they've been shrunk, compressed or trimmed. Every match comes with the exact seconds, side-by-side frames as proof, and its dollar value.

Then Claude reviews the delivery inside fixed rules. It decides what to hold, explains every decision in plain words, writes a note to each affected contributor, and sends anything uncertain to a person instead of guessing. If a contributor disputes a decision, Claude looks at the actual frames and replies with the evidence. Only a person can release a held payment, and every step, by the rules, by Claude or by a person, is logged.

How we built it

Provena Gate runs on our own open-source matching engine and dataset index (which we started earlier); the whole app on top of them was built at Cal Hacks.

  • Matching engine: every clip becomes PDQ image fingerprints (Meta's open perceptual hash) at 4 frames per second; the video itself is never stored. A multithreaded C++ kernel compares a clip against every indexed frame and votes on the time offset between two videos, so it finds the exact matching seconds. It searches 163 million frames in about 2 seconds on a laptop, holding the whole index in 2.5 GB of RAM.
  • Dataset index: five public AI video datasets (OpenVid-1M, LLaVA-Video-178K, Vript, ShareGPT4Video, VideoChat3), 1.63 million clips and ~11,500 hours, fingerprinted on fleets of AWS spot machines. Fingerprinting all of OpenVid-1M and comparing every clip against every other cost about $10 of compute.
  • The app, in Python, FastAPI and SQLite:
    • Delivery check: finds the four kinds of findings and values each in dollars, counting every second only once.
    • Rules: plain code with fixed limits (weak or still footage always goes to a person, a cap on automatic holds, overbilling detection). The AI can't override them.
    • Claude agent (Claude Opus 5.5, tool use): reads the findings and evidence and decides within the allowed actions (a forbidden action is refused and logged), then writes the buyer report and contributor notes. If Claude is unavailable, the rules' decision stands.
    • Disputes: Claude sees the side-by-side frames as images and answers with a structured reply. It can uphold or escalate, never pay out.
    • Evidence: frame strips of the buyer's own clips, and public-dataset frames fetched from Hugging Face on demand.
    • Dashboard: one page with the invoice split, findings, frames, Claude's review, disputes and the full decision log.

Challenges we ran into

  • Still footage matches everything. A colour-bar test clip matched 73 public videos. We added a check for footage that barely moves: those matches are marked weak, collapsed to one per kind, and always go to a person.
  • Keeping the AI accountable. We wanted Claude to make real decisions without ever touching money directly. Every action goes through the rules, the payment split is always recalculated in code, and every step is logged.
  • Not double-charging. One clip can match several sources over the same seconds (a dataset can hold both a video and a scene cut from it). We count every second once, and a decision on one match covers its twins.
  • Privacy between buyers. Catching footage sold twice means comparing buyers' fingerprints without showing either buyer the other's footage, names or timestamps.
  • Scale and cost. Long videos are decode-bound; we learned to keep every CPU core busy and to estimate cloud cost from hours of footage, not gigabytes.

Accomplishments that we're proud of

  • A real catch in the demo: a shrunk, compressed clip found among 1.63 million public clips, at the exact seconds, with frame-by-frame proof.
  • In a live dispute, Claude noticed from the frames that a clip described as a "city timelapse" was actually an indoor studio scene, and explained it politely.
  • 13 of 13 edited test clips found across all five datasets, with 0 false matches on unrelated video.
  • 16 automated tests, including a fake Claude that tries forbidden actions and an API outage.

What we learned

  • The hard part of AI agents in finance isn't the model, it's the guardrails: what it may do, what it must hand to a person, and a log anyone can check.
  • Exact, verifiable numbers beat big claims: "155.6 hours, checked by eye" convinces more than "huge problem".
  • At dataset scale, cost is an engineering problem: keep every core busy and measure before scaling out.

What's next

  • Catch crops, mirrors and logo overlays with a second, edit-proof visual fingerprint.
  • Pilots with video data marketplaces and collection vendors to measure their real duplicate and fraud rates.
  • More of the 300 surveyed datasets indexed, logins, and a job queue for very large deliveries.

Built With

Share this project:

Updates

Submission history