algorithm.reviews — The Agent Trust Layer

A governed AI agent that fact-checks any claim against the live web, shows every source it admits or rejects in real time, and ships a cryptographically signed receipt you can verify.

Comment

Inspiration

AI answers and online reviews are unauditable. You can't tell a grounded verdict from a hallucinated one; sources are stale, cherry-picked, or fabricated; and the "AI fact-checkers" meant to fix this are themselves black boxes. Anyone who has to defend a conclusion — a buyer doing due diligence, a journalist, a compliance or procurement team — is stuck with a vibe and ten open browser tabs.

We build trust infrastructure: a two-plane system where a governance plane decides what's allowed and an execution plane acts, fail-closed, with tamper-evident receipts. algorithm.reviews is that exact pattern pointed at a problem everyone has: is this claim actually true, and can I prove how I know?

What it does

Paste a claim, a "best X of 2026" listicle, a vendor pitch, or a URL. A governed agent:

  1. Decomposes the input into atomic, checkable claims.
  2. Searches the live web for each claim (Nimble Search).
  3. Admits or rejects each source — fail-closed. A deterministic policy rejects stale, low-authority, off-topic, or unsourced material and shows you why, live, in a green/red glass box. No admissible evidence ⇒ "unverifiable", never "false".
  4. Extracts the admitted sources (Nimble Extract) and adjudicates a verdict grounded only in admitted evidence — with per-claim confidence, the strongest counter-evidence (dissent), and verbatim citations with timestamps.
  5. Signs a review receipt with an ECDSA P-256 key. Anyone can verify it against the published public key — change a single character and the signature fails.

The result reads like an audit, not a chatbot answer: a verdict you can defend.

How we built it

  • Next.js 16 (App Router) + React 19 + Tailwind v4 + TypeScript on Vercel.
  • Vercel AI SDK 6 driving Anthropic Claude — Opus for adjudication, Haiku for source classification and query generation.
  • Nimble as the live-web layer (Search + Extract) behind a swappable WebProvider interface, with a keyless Wikipedia fallback and a deterministic mock so it builds, tests, and demos offline.
  • Two planes, one app: an execution plane (search/extract) and a governance plane (a deterministic, unit-tested admissibility policy + the adjudicator). The model proposes signals (authority, relevance); policy, not the model, decides admission.
  • Web Crypto ECDSA P-256 signing over a canonical payload that excludes volatile fields, so a re-run only changes the signature when the evidence changes.
  • A streaming "glass box" of newline-delimited JSON events, hard caps in code to stay inside rate limits and the serverless budget, and 19 unit tests (governance, canonicalization, signing, UI reducer).

Challenges we ran into

  • Web search doesn't work from a datacenter IP without a keyed provider. DuckDuckGo blocks Vercel's IPs and Jina's search now needs a key — exactly the problem Nimble's residential infrastructure solves. We added a keyless Wikipedia fallback so the app is never dead, and Nimble is the real upgrade.
  • The governance plane was too honest at first — it correctly rejected irrelevant sources and returned "unverifiable", which exposed a search bug (raw sentences make a search index return junk). The fix was LLM keyword query generation, not loosening the policy. A fabricating fact-checker would have "found support" in that junk; ours refused.
  • "Signed" has to mean signed. A bare hash is a checksum, not a signature. We implemented a real ECDSA keypair and a public verify endpoint, so a skeptical engineer can actually check it.
  • Serverless ephemeral keys broke verification until we pinned a stable signing key.

Accomplishments that we're proud of

  • A fact-checker that refuses to rubber-stamp — fail-closed, with a visible admit/reject trail.
  • Receipts that are independently verifiable and tamper-evident: flip a verdict or a quote and the signature fails, live.
  • Real verdicts on live data — true claims → supported, false claims → refuted, thin or conflicting evidence → unverifiable.
  • One build that legitimately stacks three challenges without feeling forced.

What we learned

Make the governance the product, not the answer — the admit/reject reasoning is what makes a verdict trustworthy. Determinism is a feature: a signature that only changes when evidence changes is a liveness proof you can demo. And graceful degradation (swappable providers, mock mode, offline fixtures) is what lets a live demo survive contact with a stage.

What's next for algorithm.reviews — The Agent Trust Layer

Nimble Crawl/Map for deep multi-page evidence and per-source provenance scoring; a callable MCP server so other agents can use the verifier as a fail-closed gate; shareable receipt permalinks and an org-level policy console with tunable admissibility thresholds; and a "watch this claim" mode that re-verifies on a schedule.

Built With

Share this project:

Updates