AuditBench Screener

• 

  Landing page: verdict per EU AI Act Article, with v2 re-test numbers

• 

  A hidden Article 14 human-oversight gap rated likely non-compliant, with quotes checked against the text

• 

  A fake 'pre-approved by the EU AI Office' note is flagged and ignored; the real gaps are still found

• 

  Why not one Ultra call: about 4x cheaper, abstains to a human, verified quotes, visible manipulation

Inspiration

The EU AI Act's high-risk rules (Articles 9–15) apply from 2 December 2027. Providers and notified bodies will use LLMs to pre-screen technical documentation because there's too much to read by hand. Before building a tool, I measured whether LLM auditors can be trusted.

What I found first (AuditBench)

On AIReg-Bench (120 documents labelled by legal experts):

• Good single-question auditors agree with the experts about as well as frontier models do (QWK ≈ 0.86–0.88, statistically tied).
• Unsupported "certified compliant" or "pre-approved by the AI Office" text didn't fool them (0 false clearances).
• An adaptive rewrite attack by Claude Sonnet succeeded on only 1 of 32 documents.
• The real blind spot is long files. When a violating section comes first in a full multi-Article technical file, the compliant sections after it mask it: auditors falsely cleared up to 15% of non-compliant documents.

What it does

AuditBench Screener turns those findings into a tool. Paste a technical file:

1. It judges one named Article at a time, with a prompt that ignores claims and notes to the reviewer and counts a disclosed gap against compliance. Files with "Article N" headings are split by section, so quoted evidence is easy to check.
2. Nemotron 3 Nano judges each section three times in parallel. If the samples disagree or sit on the fence, Nemotron 3 Ultra is asked; if Nano and Ultra still disagree, the Article goes to human review instead of getting a guessed verdict.
3. Text addressed to the reviewer and unsupported compliance claims are flagged and ignored as evidence.
4. Evidence quotes are only shown if they appear (near-)verbatim in your document, so hallucinated quotes are hidden.

How we built it

• Python engine (screener/engine.py) with a FastAPI backend and a no-build single-page UI.
• All model calls go through Nebius Token Factory's OpenAI-compatible API. Nano handles the fast parallel samples and Ultra the hard cases, which keeps cost at about $0.009 per five-Article file (~10–20 s per file).
• The same repo holds the AuditBench evaluation (5,760+ scored calls across auditors, perturbations, dossier stress tests, an adaptive attacker, and bootstrap CIs).

Results on Token Factory

Each expert-labelled file is placed first in a 5-section technical file (the case where generic whole-file judges miss violations). All 120 files, three repeat runs, run on Token Factory:

Re-testing our own design

After the first version we pre-registered a stricter test (protocol committed before any run): a held-out split by system, stronger baselines, repeat runs, bootstrap CIs, files with headings removed, and a 14-case known-answer regression suite (14/14 pass). 1,560 scored runs. What it showed:

• The Screener beats a whole-file Nano judge by a wide margin, but over all 120 files it is a statistical tie with one whole-file Nemotron 3 Ultra call, and it rejects more compliant files. Its case is smaller models plus more caution, not higher accuracy.
• The gain comes from the judging prompt, not from splitting the file: with headings removed it still missed only 0–4% of violations. An Ultra-based section finder we built for heading-less files worked (98% of target sections recovered) but did not lower missed violations, so we left it out of the live app.
• Single runs are noisy: three runs of the same whole-file judge ranged from 43% to 52%.

The Screener sends about 1 Article in 8 to human review.

Why not just one Ultra call?

Measured on 20 random files at Token Factory list prices (results/cost.md):

• About 4x cheaper: $0.009 per five-Article file vs $0.038 for whole-file Ultra. Most calls go to Nano; Ultra is asked only when the samples disagree (38% of Articles) and reads one section, not the whole file.
• It abstains. When Nano and Ultra disagree the Article goes to a person, instead of a guessed verdict.
• Evidence is checkable. Quotes are shown only if they appear in the submitted text.
• Manipulation is visible. Notes to the reviewer and claimed approvals are flagged, not just ignored.
• The trade-off: about 2x slower per Article (6.5 s vs 2.9 s, sequential; the app runs Articles in parallel) and more false alarms.

Challenges

• Reasoning models spent the whole token budget thinking and returned empty answers; fixed by turning thinking off for judging calls.
• Most of the NVIDIA catalogue we tried was retired or overloaded, so we standardised on Token Factory's Nemotron models.
• The honest null results: the planned "harder" attacks mostly failed, which moved the focus to long files. Our own re-test then showed the fix was the prompt, not the section split we had designed around.

What's next

Per-Annex IV section coverage, PDF upload, an Inspect AI task so regulators can re-run the evaluation, and a pilot on real redacted documentation with reviewers.

Try it

• Live demo: https://auditbench-screener.vercel.app
• Code (MIT): https://github.com/Rcidshacker/auditbench

Built With

• eu-ai-act
• fastapi
• nebius
• nebius-token-factory
• nemotron
• nvidia
• python
• vercel