PolicyOS — Verify the Policy Before You Act

Check a policy claim, inspect the source passage, and take the evidence with you.

Comment

Inspiration

A policy headline can be right about the topic and wrong about the detail that matters: the date, the affected businesses, or an exception. For an analyst preparing a briefing, a fluent answer is only the start. They need to see the passage behind it. Our demo opens on a (dramatised) chatbot that confidently says the EU AI Act entered into force on 1 August 2025 and, asked for a source, replies "Trust me." PolicyOS exists for that moment.

Our intended users are policy and business analysts. That is a product hypothesis: we have not yet established external adoption or measured time savings.

What it does

Enter a policy claim and watch the check run. PolicyOS separates checkable assertions, searches the live web, screens the retrieved documents and reports supported, refuted or insufficient evidence for each assertion. An optional mode looks for official sources and reads the relevant pages.

  • Evidence trail. The result shows source URLs, the exact retained passages supplied to the adjudicator, the model used at each stage, Tavily request IDs and a "What ran" trace.
  • Exports. The whole record downloads as a report or as JSON.
  • Human review. Results can be saved to a review queue as low-confidence, AI-inferred items. A verdict cannot change a country's score automatically; a person must approve it.
  • Map. /explore adds context: a directory of 217 countries and economies. Directory presence, monitored sources and curated scores are shown as separate things, and missing data is labelled as missing rather than treated as zero.
  • Policy events. /monitor groups official announcements into traceable research leads.

A recognised government domain is a navigation hint, not proof that a passage settles the claim. A source's publication date is not necessarily its effective date.

How we built it

claim
  → decompose     NVIDIA Nemotron 3 Super 120B-A12B (Ultra fallback on transport failure)
  → retrieve      Tavily Search, bounded calls
  → inspect       optional Tavily Extract, scoped official search, bounded issuer HTML read
  → screen        NVIDIA Nemotron 3 Super 120B-A12B
  → adjudicate    NVIDIA Nemotron 3 Ultra 550B-A55B
  → roll up       supported / refuted / insufficient evidence
  • Models. All Nemotron calls are runtime calls to the Nebius Token Factory inference API. Super handles the short decomposition and screening steps; Ultra weighs each assertion against the retained passages. Structured tool calls pass a defined result between stages. Our client forces the requested function and rejects missing calls, invalid arguments and truncated completions, so a transport or format failure is never presented as a policy verdict.
  • Search. Tavily is part of the live evidence path. Search finds documents for the actual assertion and Extract reads selected pages. Request IDs are retained.
  • App. Next.js on Vercel with PostgreSQL. Shared admission limits and a cumulative model-spend guard protect the hosted demo. The web UI, API and evaluation scripts share one verifier implementation. The public repository has 629 offline tests in CI.

Challenges we ran into

The hard part is preserving the qualifier through every stage: the query, source-page selection, excerpt selection and final citation. An official page can be relevant but incomplete for the specific claim. A correct refutation of one clause does not mean every clause has been verified. Provider failures, storage failures and verdicts each needed their own visible outcome.

Accomplishments that we're proud of

  • Evidence you can inspect. The evidence behind a verdict is preserved and exportable, not just summarised.
  • Honest evaluation. We publish failures alongside successes. In a small, developer-selected check, labels were right more often than the evidence was complete: two new cases gave 2/2 expected labels but only 1/2 strict issuing-body evidence passes. Later repairs on familiar cases are regression evidence, not new accuracy. See EVALUATION_SUMMARY.md in the repository.
  • Recorded hosted runs. On 13 October 2026 the hosted app completed real checks, including the EU AI Act run shown in our video: refuted, citing the European Commission's page (1 August 2024), with 3 Nemotron calls on Nebius Token Factory and Tavily Search/Extract. These are recorded functionality checks, not accuracy or latency claims.

What we learned

A correct label and complete evidence are different measurements, so we score them separately. Successful extraction can still lose the scope that determines a policy answer, such as a cohort or an exception, which is why PolicyOS keeps the exact text that reached the adjudicator.

What's next for PolicyOS

  • Broader issuer-domain discovery.
  • Context-preserving passage windows that keep nearby headings and exception clauses.
  • A larger, independently labelled evaluation set.
  • Reviewer workflows that compare a new verdict against earlier saved evidence.

What changed during the hackathon (pre-existing project)

PolicyOS existed before 26 August 2026 as a policy monitoring and scoring site. The hackathon work adds:

  • runtime claim verification with Nemotron on Nebius Token Factory and Tavily;
  • optional source inspection;
  • exact evidence exports;
  • review-aware persistence;
  • source-discovery paths on the map;
  • shared spending controls;
  • a documented live evaluation history.

About the demo video

The video (114.8 s) opens with an original pixel animation. Its chat window is dramatised and labelled "not a real model". Every product shot is a real, edited recording of a historical run on 13 October 2026 UTC, labelled as such; it is not a new run. The evidence card shown is the model's verdict summary with its linked source, which the video labels on screen. It is not a full capture of the source page text.

Music: an 8-bit transcription by Ctske of Johann Pachelbel's Canon in D (OpenGameArt.org), CC0 https://creativecommons.org/publicdomain/zero/1.0/ , with edited excerpt, fades and volume changes.

Testing instructions

  1. Sign in at https://policyos-app.vercel.app/judge with the judge testing code in the private file under this submission's "Additional info" (visible only to judges and organizers). A session lasts up to 8 hours; if it ends, sign in again with the same code. Judge access ends on 15 December 2026 at 12:00 pm PT.
  2. Open https://policyos-app.vercel.app/verify and enter a claim, for example: The EU AI Act entered into force on 1 August 2025.
  3. Enable "Seek official sources and read source pages" and click Verify.
  4. Inspect "What ran", the verdict, the retained excerpts and linked sources, then use "Download report" or "Download data (JSON)". A fresh model response is not guaranteed to match the recorded run.
  5. The dated EU and BOE recordings open without signing in and without provider calls. They are historical examples, not fresh checks.
  6. /explore (map and country directory) and /monitor (policy event leads) open without signing in. Live country search on the map also uses Tavily and needs the judge session.

Live checks make paid model and search calls, so they are open only to judge sessions; public visitors cannot use up judging capacity. Judge sessions are not subject to the public daily run limit, but all judge sessions share a fixed allowance of 120 live checks, a fixed model spending cap and the project's search credits; live country search is limited to 60 searches per day. If the page reports that an allowance, budget or search capacity is used up, it does not reopen on its own and needs the project owner; the dated recordings remain viewable but are not live checks. The video shows a recorded run from 13 October 2026, not a new run.

Built With

Share this project:

Updates

Submission history