At a glance

Try it in one click: polyjury.vercel.app/#recorded replays a finished case on DeepMind's SO-101 robot-arm model. The verdict, the measured collision and the proved patch are at the top of the screen in about 10 seconds, nothing to set up (that patch stops contact deeper than 3 mm; upstream chose 1.474 rad, the first touch). A second recorded case, a Python CLI, is one click away in the sidebar. For a live run (2 to 3 minutes), paste any public GitHub repository at polyjury.vercel.app.

What is different

  • A proof that ran, not an opinion. Every claim gets a script that runs in its own Nebius Sandbox microVM. It is CONFIRMED only when the run shows it, and Nemotron throws out runs whose output does not support their own verdict.
  • Robots by simulation. For URDF/MJCF robot files, Polyjury measures joint sweeps and mesh overlap in MuJoCo. In my runs, the three jurors never found a self-collision by reading; the simulation did.
  • Checks upstream first. Before you report anything, the repository's issues and pull requests and the web are searched, and Nemotron decides whether a hit is the same defect.

Results, including the misses

  • DeepMind MuJoCo Menagerie, SO-101: elbow self-collision reproduced (from 1.474 rad; the limit is 1.69) → issue #325, filed by me → another contributor then opened PR #332, "Fixes #325". A maintainer asked for the limit to sit right where contact starts; on 19 Oct the author set it to 1.474 rad, the onset Polyjury measured. Still open as of 23 Oct.
  • 8 public repositories, checked by hand: 11 claims reproduced → 2 real defects, 9 not defects (intended or unreachable behaviour, generic advice, a rounding difference).
  • Live run on fastapi-cli: 8 claims → 2 reproduced, 1 false alarm, 3 thrown out by the audit, 2 failed to run (a cache race, fixed since).
  • Control pair, used while building the robot check: a toy arm with a 0.6 m forearm → collision reproduced; the same arm with a 0.4 m forearm → 0 contacts.
  • Tried and turned off: three independent proofs per claim with a majority vote: no more agreement than one proof, and the slowest claim went from 92 s to 361 s.

What it can't do (yet)

  • Verdicts vary between runs on the same repository. Measured on 25 Oct, three runs of fastapi-cli in a row: 8 claims each time, and 0, 4 and 3 of them reproduced; the same claim was reproduced twice and rejected by the audit once. The robot simulator check is the stable part: 4 of 4 runs on SO-101, identical numbers, the fix proved each time.
  • "Reproduced" means the behaviour happened, not that it is a bug: 9 of 11 field reproductions were not defects. Where it only reads, Polyjury says so, or says NEEDS A HUMAN.
  • Proofs are Python scripts (plus MuJoCo for robot files), and only public GitHub repositories can be checked.
  • A full run takes 2 to 3 minutes on shared demo credits with rate limits; you can bring your own Nebius key.
  • Note for judges: on 9 Nov Nebius retires three of the models named on this page from serverless (nemotron-3-super-120b-a12b, DeepSeek-V4-Pro and Qwen3-235B). After that, the live site runs a newer Nemotron as chair, set by a config value with no code change, and the listed stand-in jurors. Every verdict names the model that stood in.

Inspiration

I vibe-code. The AI that writes my code is also the AI I ask whether it is any good, and it always says yes. Or it lists ten "potential issues" and I spend an afternoon finding out that eight of them are not real. I did not want a longer list. I wanted a review that only says something when it can show it. So the rule became: no bug report without a repro.

What it does

Paste a public GitHub repository, or one folder of it.

  1. Three open models review it separately (DeepSeek-V4-Pro, Qwen3-235B and gpt-oss-120b on Nebius Token Factory) and never see each other's answers, so one model's blind spot is not the verdict.
  2. NVIDIA Nemotron presides. It merges the reviews, drops what cannot be checked by running code, and for every surviving claim writes a Python script that tries to make the defect actually happen.
  3. Every script runs in its own Nebius Token Factory Sandbox: a disposable microVM, after an attempt to install the repository's declared dependencies (a failed install is printed, not hidden). Model-written code that tries to break things is exactly what you should never run on your own machine.
  4. Nemotron audits every run. A script whose printed output does not support its own verdict is thrown out. A script that only grepped the source is reported as a reading, never as a run.
  5. Sources. Tavily searches CWE, OWASP, MDN, PortSwigger, the RFCs and the official docs for the literature that already names the defect. Only what reproduced reaches you, with the run's own output as evidence, and a prompt to paste back to the AI that wrote the code.
  6. Already known? Before a reproduced defect is reported, Nemotron writes the queries a maintainer would type, the repository's own issues and pull requests (open and closed) and the web (Tavily) are searched, and Nemotron decides whether any hit is the same defect: known, fixed, described elsewhere, or new. A web page alone can never make a defect "known".
  7. Fix it, then prove the fix. For a reproduced defect, Nemotron rewrites the file; the same proof runs again on the patched copy and has to come out NOT REPRODUCED; then Nemotron reviews the diff for a genuine fix rather than a special case for the test. The proof it re-runs is the one the server sealed; the browser cannot hand the sandbox a script of its own. When a patch does not hold, the page says NOT FIXED instead of pretending.

And it does the same for robots. A URDF or MJCF file is code too, and a wrong joint limit ships a robot that drives itself into its own frame. When a repository holds one, the sandbox also gets MuJoCo, the meshes and a tested helper, and Polyjury adds a claim no juror can make by reading: it sweeps every joint alone inside its own limits and re-checks every contact on the exact meshes. The fix for a robot is measured, not generated: the first-contact angle minus a margin, with the lost travel stated in degrees.

The numbers

  • SO-ARM100 (the open-source arm used with LeRobot): in so101_new_calib.urdf, elbow_flex alone drives gripper_link into shoulder_link from 1.51 rad (convex hulls) / about 1.49 rad on the exact triangles. The file allows 1.69 rad, and at that limit the exact overlap is about 12,400 mm³. None of the three jurors found it by reading; the simulation did. An open pull request on that repository, measured with Drake on adjacent links only, proposes 1.573 rad, which is still inside the collision; I posted the measurement there. The fix loop tightens the limit to 1.4901 rad, re-simulates clean, and reports 11.5° of travel lost, all in 39 seconds on the live site.
  • In the field. I ran Polyjury on 8 public repositories I had no hand in: 3 robot-arm descriptions (MuJoCo Menagerie among them) and 5 small Python projects. 11 claims reproduced. Checked by hand against the source, 2 were real defects and 9 were not defects: intended behaviour (a fallback that treats a score of 0.0 as 1.0 by design), a KeyError on a path the code can never take, generic advice, a rounding difference. Both real ones were in DeepMind's mujoco_menagerie SO-101 model: the elbow self-collision again (from 1.474 rad, 24.6 mm deep at the limit), now filed as issue #325 (another contributor has since opened pull request #332 to fix it, reproducing the same 24.6 mm), and a wrist actuator whose range disagrees with its joint. That second one I first read as a typo in the actuator; a pull request opened the day before, and the source URDF, say the joint range is what drifted. A mismatch only tells you two numbers disagree; the source tells you which one moved. That check is now a rule.
  • Polyjury on Polyjury. Run on its own code, it reproduced a cross-site-scripting hole in its own page (the escaper only stripped < and >, so a source title could break out of an attribute) and the fact that only one of its endpoints had a rate limit, so the others could drain the demo's credits. Both are fixed. The jurors also missed one that a human found while reading their output: the proof endpoint accepted a server-local path. That is fixed too.
  • Three models produced 17 findings on this project's own code and only 2 of the merged claims were raised by more than one model. A single reviewer misses most of it.
  • A live verdict on fastapi-cli (in the video) takes about two and a half minutes: 8 claims, each proved in its own sandbox. The result: 2 reproduced, 1 false alarm, 3 whose scripts contradicted themselves and were thrown out by the audit, and 2 the site failed to run at all (a cache race between parallel proofs, fixed since). The fix loop then patched its malformed-TOML crash and re-proved it in 48 s; a second claim survived two patch rounds and is shown as NOT FIXED.

How I built it

Python and FastAPI on Vercel. Nebius Token Factory's OpenAI-compatible API for the jurors and for Nemotron (nvidia/nemotron-3-super-120b-a12b, thinking enabled for merging, proof-writing, auditing, patching and diff review, which is judgement, not retrieval). The Sandboxes SDK for the proof runs. MuJoCo + trimesh + manifold3d inside the sandbox for robot descriptions. Tavily for the sources and for the already-known check. There is also a single-file MCP server, so Claude Code or Codex can ask Polyjury for a verdict and fix only what was confirmed. Sign-in with GitHub is optional and asks for no scopes, and you can bring your own Nebius key instead of the demo credits; nothing is stored server-side: the token and key live only in encrypted httpOnly cookies in your browser, and signing out revokes the token at GitHub (policy).

Challenges I ran into

  • Getting a script out of a thinking model reliably. JSON with code inside breaks on newlines, so proofs come back in a code fence and are parsed with ast before they run; one that will not parse is sent back once for repair.
  • Proofs that "reproduced" nothing. A script that prints a precondition (two files share a name) and calls it REPRODUCED. The audit now rejects precondition-only evidence and errors caused by the script's own setup. It also threw out two claims from my own earlier demo, and I kept it that way.
  • Proof scripts contaminated the next review. A script that planted an attack module left it behind, and the following review read the attack as part of the project. Every proof now runs on a throwaway copy.
  • A clean microVM has none of the project's packages. Half the early proofs were inconclusive because a fresh Python image cannot import typer. Each proof VM now tries to install what the repository declares in pyproject.toml or requirements.txt (plain requirement strings only), plus ffmpeg and fonts when the code calls them; a failed install is printed in the output, not hidden.
  • Models disappear. One juror answered 404 for an afternoon without notice. A stand-in list now takes the empty seat, and the verdict says who stood in for whom.
  • Proofs start in parallel, and two of them once unpacked the same repository into the same cache folder at the same time. Found during the final recording, fixed the same night.
  • Majority voting did not help. I tried three independently written proofs per claim, expecting steadier verdicts. Measured on the same eight claims twice: no more agreement than a single proof, and the slowest claim went from 92 s to 361 s. The instability comes from claims that can be read two ways, not from the luck of one script, so the feature is off, and the control run in the fix loop (the same sealed script must reproduce again on the untouched code before anything is patched) does the honest work instead.

What I learned

"Reproduced" means the behaviour happened, not that it is a defect: 9 of the 11 field reproductions were not defects on a closer look, and a proof that shows a CLI importing the module its config points at has reproduced something by design. The audit can check consistency; deciding what is a bug still needs the reader, and Polyjury says so with NEEDS A HUMAN instead of pretending. Where the tool measures (a joint angle, an overlap volume, a range that disagrees with its source), the verdict holds up; where it reads, a person has to.

What's next

Robot descriptions are the first kind of hardware Polyjury can prove things about, and the measuring part has since become a standalone, model-free tool with a GitHub Action, jointsweep, that Polyjury will depend on. A first pre-registered pass over the 25 arms listed in Menagerie's README (25 Oct, jointsweep, no models, under two minutes) flagged 19. One is confirmed, the SO-101 elbow, already reported; the other 18 have not been reviewed by a person yet. Fingers that close on each other are flagged too, and "home" as all-zero joints is outside the range on some arms, so the next step is the human review and a better home pose, not more flags. Next: report only what a person has checked; and a GitHub Action that comments on a pull request only with what reproduced. The already-known check (step 6) now runs on every reproduced defect. On the live site it found pull request #332 for the elbow, and for the wrist actuator it finds the pull request that would have caught my own misreading.

Built With

  • claude
  • claude-code
  • deepseek
  • fastapi
  • gpt-oss
  • nebius
  • nebius-token-factory
  • nvidia-nemotron
  • playwright
  • python
  • qwen
  • tavily
  • ttapp
  • vercel
Share this project:

Updates

Submission history