Sayproof

Napoleon? Not short. Einstein? Never flunked math. Vikings? Zero horns. Sayproof checks every claim in your rehearsal, brings the receipts, and hands you a better line. Stay chill, stay right. 🤙

Comment

Inspiration

In 2003 I was a student preparing for an exam, and I had learned what the textbooks said: neutrinos are massless. Except that since 1998 they weren't. Super-Kamiokande had shown that neutrinos oscillate, so they must have mass (a Nobel Prize in 2015). My notes were five years out of date, and I had no idea. That is the kind of mistake Sayproof is built for, and it would have rescued me. Today it marks "Neutrinos are massless particles" contradicted and quotes its source: "Although neutrinos were long believed to be massless, it is now known that there are three discrete neutrino masses …" Then it gives me the line to say instead: "Neutrinos have tiny, non-zero masses."

Nobody who says something like this is careless. Knowledge moves on without telling us. "Pluto is our ninth planet" was true until 2006; Sayproof marks it outdated, not wrong. And sometimes every number is right, but the picture isn't: "From the Big Bang to the edge of our universe it is 46 billion light-years" leaves the audience imagining an edge, when 46 billion light-years is how far we can see. Sayproof calls that misleading and explains why.

So we didn't build a pedant that counts decimals (spoken rounding is fine: "about eight minutes" for 8 min 20 s). We built a friendly rehearsal partner that catches what changed, the myths everyone repeats, and the numbers that paint the wrong picture, so speakers can walk on stage, or into an exam, knowing they're right.

What it does

You paste the text of your talk, or record a rehearsal. Sayproof picks out the checkable claims, searches for evidence, and gives you a receipt. Each claim gets one of five verdicts, and every verdict comes with a sentence quoted from a source:

  • Supported: a source confirms it.
  • Contradicted: a source shows it's false, with a sentence you could say instead.
  • Outdated: it used to be true ("Pluto is our ninth planet" was true until 2006).
  • Misleading: the number is right, but the picture is wrong. "From the Big Bang to the edge of our universe it is 46 billion light-years": 46 billion light-years is how far we can see, not an edge. The verdict has to name the false belief the audience would take away, and it has to survive a second, reasoned look.
  • Can't verify: the sources don't settle it. This includes claims about you or your company ("our users tripled"): no public source can back those, so Sayproof asks you to have your own data ready instead of guessing.

When you record, on the live site or locally, each claim on the receipt has its time in the talk. Click it to hear that moment again. Talks up to 30 minutes work, and verdicts stream in as they finish.

The rule the whole product rests on: no source, no verdict. If the model can't quote a sentence that's actually in the source text, the claim becomes "can't verify".

The receipt starts with what to do ("Fix 2 claims before you give this talk"), and each wrong claim starts with the sentence to say instead. The quote and the source come right below it, for anyone who wants to check.

Why this isn't a generic "paste text, get a fact-check" wrapper. A speaker's problem is different from a reader's, and that shaped every part of the product:

  • It works on speech. Record a rehearsal, and every claim on the receipt links back to its second in the recording.
  • "Wrong" and "no longer true" are different fixes. The outdated verdict tells you to update a number, not to drop the point.
  • It knows what it can't check. Claims about you ("our users tripled") are never searched, because search finds someone else's numbers for almost anything.
  • Reference sources come first. A fact-checker that cites trivia sites repeats the myths a speaker wants to avoid. The open web is a labelled fallback, not the default.

How we built it

audio ─► NVIDIA Parakeet (local, MLX) or ElevenLabs Scribe (live site): transcript + a time for every word
text  ─► NVIDIA Nemotron 3 Super on Nebius Token Factory: checkable claims, in parallel chunks (talks up to
         30 min, up to 40 claims), the speaker's exact words kept, claims about the speaker flagged
      ─► Tavily, tier 1: reference sources only (public bodies, science organisations, encyclopedias, major news)
      ─► NVIDIA Nemotron 3 Ultra, reasoning: verdict + evidence sentence copied word for word
      ─► quote guard: is the sentence really in the source? one retry, then "can't verify"
      ─► only if still unverified: Tavily tier 2, open web with user-generated and video sites blocked
      ─► "misleading" gets a second reasoned look; wrong claims get a "say instead" line rewritten for the ear
      ─► receipt, streamed to the page verdict by verdict
  • Nebius Token Factory serves two NVIDIA models through its OpenAI-compatible API. nemotron-3-super-120b-a12b extracts claims fast, in parallel. Nemotron-3-Ultra-550b-a55b with reasoning writes the verdicts: telling "the number is right" from "the picture is right" needs judgement.
  • The judge sees the speaker's words, not a tidied claim. Our extractor used to "helpfully" rewrite "the edge of our universe" to "the observable universe", which removed the very error. Now extraction must not correct anything, and the judge reads the original sentence.
  • NVIDIA Parakeet (parakeet-tdt-0.6b-v3, 25 European languages including German) transcribes recordings locally. It gives a time for each token, which we map back to every character, so a claim's position in the text becomes a time in the recording. On the live site, ElevenLabs Scribe does the same job, because Parakeet isn't hosted on Token Factory.
  • Tavily is the evidence layer, in two tiers. Tier 1 is restricted to a list of reference domains. Only if that leaves a claim unverified do we search the open web, with a blocklist, and the receipt labels that source "outside the reference sources".
  • App: FastAPI and one static page on Vercel. Recordings go from the browser straight into a private Supabase Storage bucket through a signed URL (past Vercel's 4.5 MB request limit); ElevenLabs reads them through a 10-minute link. Rate limits protect the API credits on the public demo.
  • The pitch video was made with the same kind of open models: LTX-Video on a Modal H100 for the space flight behind the opening, FLUX.1-schnell on Modal for the thumbnail background, ElevenLabs for the voice. Every result shown in it is a real Sayproof output.

Challenges we ran into

  • True numbers, wrong picture. "46 billion light-years to the edge of the universe" is a correct number, and a plain fact-checker says "supported". The audience still leaves believing the universe has an edge. We added a "misleading" verdict, but at first it fired on true, carefully hedged claims too (2 of 20 true TruthfulQA answers). Requiring the verdict to name the false belief, plus a second reasoned look, brought that to 1 of 20.
  • Telling "false" from "no longer true". For a speaker, "outdated" is a different fix from "wrong", so the verdict prompt defines it with examples ("was", "until 2006", "no longer"). Claims that name their own date ("in 2005, Pluto was …") are judged for that date.
  • Models paraphrase quotes. A verdict is only worth something if its quote is real, so we check it against the source text. One retry that asks for an exact copy fixes almost every paraphrase: in our last evaluation (150 verdicts), no final verdict had to be downgraded.
  • Source quality. On the open web, fewer than half of the cited sources (16 of 36) came from reference sources. Restricting Tavily to a reference list raised that to 33 of 33 but missed some lesser-known facts (33 of 36 correct). The two-tier search kept both: 36 of 36 correct, 35 of 36 citing a reference source.
  • "Our revenue doubled" came back "supported", with a quote about a different company's revenue. Search finds someone's numbers for almost any claim. Claims about the speaker are now flagged at extraction and never searched.
  • Respectable pages repeat myths. The classic presentation myths (goldfish attention span, "93% of communication is nonverbal", the 10%/90% learning pyramid) appear on university and news pages that mention them in passing. Sayproof took those mentions as confirmation and was confidently wrong in 6 of 15 runs. A prompt rule that a statistic repeated in passing doesn't confirm it, only the original measurement does, brought that to 1 of 15.

Accomplishments that we're proud of

  • An evaluation that runs the production path, not a simplified one: 55 claims with a known answer (true facts, myths, presentation myths, wrong numbers, outdated claims, dated claims, claims about the speaker; 9 in German), 3 runs each.
    • 165 of 165 verdicts match the label or a defensible alternative (for example "contradicted" for an outdated claim). 158 of 165 match exactly.
    • 0 of 165 confidently wrong, meaning a wrong verdict other than "can't verify". This is the error that hurts a speaker most. Earlier runs had 2 and 3.
    • 51 of 55 claims got the same verdict in all 3 runs. Median 4.0 s per claim.
    • In 4 sample talks, all 30 checkable claims were found and none of the 27 opinions, greetings or plans were kept.
  • On TruthfulQA, a public benchmark of common misconceptions (each question gives one true and one false answer to check), with the current system: a 40-claim sample came back 36 right, 3 "can't verify", 1 confidently wrong. The full run is harder and more honest: every usable question (386 of 817, 29 categories) gives 594 of 772 claims right (77%), 78 "can't verify", 100 confidently wrong (13%), in 21 minutes. In the "Misconceptions" category, the closest to what speakers repeat on stage, it's 109 of 122 right (89%). Many of the misses are claims no speaker would make ("broomsticks can sweep away bad luck") or comparisons the sources don't state directly. That full run is now our benchmark for the next round of fixes. (The 165-verdict evaluation above predates the "misleading" verdict and the Ultra judge.)
  • Every design choice was measured against the simpler alternative (these measurements are from the Nemotron 3 Super version, before the Ultra judge):

| Choice | Simpler alternative | Sayproof | |---|---|---| | Where to search | Open web: 16 of 36 citations from reference sources | Two tiers: 35 of 36, same correctness (36 of 36) | | Reasoning on verdicts | Always on: 11.1 s per claim, 3 of 165 confidently wrong | Only for "outdated": 4.0 s, 0 of 165 | | Presentation myths (goldfish attention span, 93% nonverbal, …) | Confidently wrong in 6 of 15 runs | 1 of 15, after a rule that a statistic repeated in passing doesn't confirm it |

  • Every verdict on a receipt can be traced to a sentence in a source the user can open.
  • Speech input that maps every claim back to its moment in the recording.

What we learned

  • Measure before choosing. The source-filter decision came from a comparison on labelled claims, not from intuition. Tavily's relevance score, for example, barely filters anything for reliability.
  • Small evaluation sets still find real bugs. All three fixes in the last round (claims about the speaker, rounded numbers, myths repeated by reputable pages) came from reading the misses one by one.
  • "Can't verify" is a feature. A checker that admits it doesn't know is more useful than one that always has an answer.

What's next for Sayproof

  • Host Parakeet next to the app (Nebius AI Cloud), so the live site runs on NVIDIA speech too, not only locally.
  • Judge estimates as ranges ("billions of galaxies" vs. "quadrillions"), and run the full TruthfulQA set (817 questions) on every prompt change.
  • Delete stored recordings after 30 days.
  • Check slide decks (PDF) and speaker notes, not only spoken text.
  • A larger evaluation set, with fresh searches on every run, to measure search noise as well as model noise.
  • Re-check published material on a schedule: the same engine can flag statements on your own website that have gone out of date.

Built With

Share this project:

Updates

Submission history