PRISM

A systematic review of how public AI launches evidence their readiness: Nemotron reads each project's own site, a deterministic gate decides, and the coding's accuracy is measured.

Comment

Judge test access

Inspiration

Hundreds of AI products launch on Show HN every month, each claiming to be ready for something. Buyers, reviewers and the builders themselves have no common way to ask: ready for which decision, against which standard, and backed by what evidence? PRISM answers that the way a systematic review paper would, for a whole sample of launches rather than one.

What it does

PRISM is a checklist for users, a tour guide for builders and a benchmark for investors. Users get a checklist of the evidence to look for before trusting an AI launch, and how often launches actually show it. Builders get a private report showing where they stand and the next piece of evidence to add. Investors get a benchmark of how launches evidence readiness against ISO/IEC 42001, the NIST AI RMF, the EU AI Act and the OECD AI Principles.

PRISM is not a promotion or listing service: nobody can pay, submit or apply to be included, ranked or recommended. Projects are sampled from public launches, evidence comes only from each project's own site and repositories (never directories or listings), per-project results stay private, and the public sees aggregates only.

PRISM (Public-evidence Review of Innovation, Standards & Markets) discovers self-promoted AI projects, gathers evidence only from each project's own website and repositories, and codes it against a codebook that joins the ai-output-to-value decision gates with ISO/IEC 42001, the NIST AI RMF, the EU AI Act and the OECD AI Principles. Every coded value is backed by a quote or recorded as unknown. A deterministic, open gate (not the model) decides each claim. The public output is aggregate statistics and a written review. Per-project analyses are private: judges and each company can see them in a signed-in app, which can also run PRISM live on one project.

I reviewed 300 AI launches from Show HN with hypotheses fixed in advance. At least 169 publicly evidence their intended use and 162 state their limitations, but just 44 show a fallback and 104 the operating-cost boundary. Both pre-registered hypotheses were not supported, and I report that. Of 24 checks, four are released with counts, three are shown as fewer than ten, and 17 are withheld because their coding did not meet the pre-registered accuracy gate. Nemotron Ultra changed 634 of 1,610 proposed positives on confirmation.

How I built it

I built and proved PRISM with NVIDIA Nemotron models served by Nebius Token Factory, and with Tavily. On Token Factory I tested 12 Nemotron configurations (377 calls, $1.22). On Token Factory I tested 12 Nemotron configurations (377 calls, $1.22). Nemotron Lightning screened 1,675 Show HN launches down to 300 AI products, Nemotron Ultra confirmed the pilot's findings, and an open model coded all 300 projects. Tavily gathered each project's evidence from its own site, for 142 projects, without leaving the project's domain.

To scale to the full sample, PRISM is built for interoperability: each model role and the evidence collector are swappable providers. The same Nemotron models run on another host for bulk validation, and a free own-site collector took over from Tavily. In an exploratory comparison on those 142 projects, the two collectors gave the same check result 91.5% of the time. Because decisions come from the gate, not the model, a provider can be swapped without changing what counts as evidence.

The app is TanStack Start and React, with better-auth roles (admin, judge, company), SQLite, and a gate-first report for each project. It runs on its own server with HTTPS.

Challenges I ran into

  • Keeping collection on the project's own site: under default settings, 37 of 50 pages collected in my first trial came from other sites.
  • Cost control: one early run sent whole pages as quotes and used far more tokens than planned. I added a per-request token cap and a per-run budget that stop a run before it overspends, and fixed the quotes.
  • Prompt injection from collected pages: flagged text can never count as confirmed evidence.

Limitations and failure cases

  • Nemotron Lightning stopped following the JSON schema with thinking on (4–8% valid), so screening runs with thinking off.
  • Even the best setting is not perfect: in a 50-call repeat of Ultra with json_schema, the longest run of consecutive valid replies was 23. Every reply is validated, retried once, then recorded as "unknown".
  • One early confirmation run sent whole pages as quotes: about 17M input tokens and $18.30 before it was stopped. Its results are archived, never published, and the run led to the per-request token cap and per-run budget now in front of every paid call.
  • Screening depends on how the model reads a title: in tests on another host, the same project was screened in on one server and out on another. Screening is audited (n=50 sample; error rate 12.2%).
  • The free collector and Tavily agree on 91.5% of check results over 142 projects, but they share only 44% of pages: an exploratory figure, not a validation.
  • Collection recall against a separate gatherer is 0.887 on 10 projects.
  • Repeated live runs on the same project vary slightly: in 10 test runs, the one or two positive checks came out as pass in some runs and as claimed in others. The gate is deterministic over what it is given; the model's coding varies.
  • When evidence is thin, the gate says "insufficient evidence" instead of guessing, so many verdicts are "insufficient evidence": a finding about public evidence, not a verdict on the product.
  • Reddit was not collected automatically (its Data API needs pre-approval). The live app does not render pages that need JavaScript.

Accomplishments that I'm proud of

  • An open, deterministic gate that makes every decision reproducible from the evidence record.
  • Measured accuracy instead of assumed accuracy: an independent second coder and gold labels from a model family that neither codes nor confirms.
  • Private per-project results with role-based access, and public results that are aggregates only.

What I learned

A model is a good reader and a poor judge. Letting Nemotron quote the evidence and letting an open gate decide made the results checkable and the providers swappable.

What's next

Publish the aggregate review and findings site, extend the sample, and open the private report to each company so it can check and correct its own record.

Built With

Share this project:

Updates

Submission history