ChatProofs

Prove/Disprove Your Gut. Improve Your CashFlow

Comment

Inspiration

I've believed for a long time that humans and agents working together produce more leverage than either alone, and that the same is true across model vendors. I found that 79% of major enterprises pay for both Claude and ChatGPT. Most people read that as hedging. I don't think it is. I think they're buying something they haven't named yet.

I wanted to find out whether that was true or just a nice story, so I built something that depends on it being true.

ChatProofs turns a hunch into a proof you can check.

Here's the trick: two AIs from two different companies, and they can't see each other's work.

  • Claude builds the best honest case for your hunch and finds real sources
  • GPT-5.6 — a different vendor — sees only the claim and the sources. Never Claude's answer, never how confident it was
  • The score comes from whether they agree, not from either one grading its own work
  • If they disagree, it says so out loud instead of averaging them into a comfortable middle

Every claim gets a plain-English label:

  • "A source backs this up" — with a link you can click
  • "That's a fair leap" — reasoning, not proof, and it says so
  • "Someone needs to prove this"
  • "This is actually contradicted"

And one rule the software enforces on itself: it physically cannot mark a claim source-backed without a real citation attached. That's built into the data structure, not a polite instruction in a prompt.

Real example — a $1 price a company published about itself:

  • ✅ "A price of $1 is declared" — accepted, 95% confident
  • ❌ "Someone actually paid that $1" — contradicted, 5% confident

It gave exactly the credit the evidence deserved and not an inch more.

How we built it

Codex built Relay, the Formalizer — the component that turns an intuition into a typed, cited graph. Session 019f777a-b06d-7952-bb34-09de005381e5. It defined the contracts (claims, citations, assumptions, limitations, and typed edges: supports, qualifies, connects, depends_on), implemented the server-only adapter with a strict JSON schema, wrote atomic persistence and the endpoints, and wrote the tests. It also wrote its own build log, which is in the repo.

GPT-5.6 does two jobs. It's the independent verifier in the assessment pipeline. And in Relay it runs with web search and a schema where every citation requires a URL — instructed that every URL must be a page it actually found. That turns "don't make up sources" from a request into a constraint.

The engine runs on Cloud Run with the keys in Secret Manager and rate limits on every route that spends money.

Challenges we ran into

The first live run returned the same verdict for three test cases carrying completely different evidence. Nothing errored. It turned out the model was wrapping its JSON under the schema name, the normalizer read the wrapper, found nothing, and quietly substituted defaults. So both "independent" judges were returning constants while looking perfectly reasonable. That's the worst failure a proof engine can have, and I only caught it because three very different inputs produced identical output.

The second one was worse, because I'd already written the conclusion down. With gpt-4.1 as the verifier, the two vendors disagreed wildly and I concluded that cross-vendor verification adds noise. That was wrong. I'd picked gpt-4.1 defensively and generalized from the weakest model available. With GPT-5.6 as the verifier, the two vendors agreed on every case. Independent isn't the same as rigorous — the second opinion also has to be good.

Accomplishments that we're proud of

It wouldn't certify my own claim about my own product. I asked it to assess AgentBooks — my company — and it returned insufficient_evidence and told me why.

GPT-5.6 also found a real flaw in work I'd already shipped: nothing tied each payment to a service request, completed work, an invoice, or a recognized revenue event. That's the difference between cash receipts and revenue. It changed my roadmap.

And the whole thing, including both times I was wrong, is written down in a log rather than quietly fixed.

What we learned

Verifier quality dominates. A weak second opinion is worse than none, because it looks like verification.

When one model checks itself, agreement feels exactly like validation and means nothing. The engine labels that same_brain, and it should.

The dangerous failures are silent. Nothing crashed in either bug. Confident wrongness presents as competence, so you have to construct comparisons that ought to diverge and treat "they didn't" as a bug rather than a result.

What's next for ChatProofs

Relay becomes a standalone Formalizer agent that supplies evidence and renders no verdict, so the judging layer stays independent of whatever produced the evidence.

Beyond that: distinct-payer analysis so on-chain receipts can be shown to be arm's length, a published recomputation recipe so any figure can be independently reproduced, and the payment-to-recognition link GPT-5.6 told me was missing.

Built With

Share this project:

Updates