ProofLayer

• 

  Three live runs with Nemotron on Nebius Token Factory.

• 

  All 18 benchmark cases got the expected verdict.

• 

  85 tampering attempts after issue, all caught with the detached ledger.

• 

  How ProofLayer works, from agent to certificate.

• 

  Certificate for a live NVIDIA Nemotron run, with every limitation listed.

• 

  The agent said its tests passed. The independent verifier says otherwise.

• 

  A proof challenge: one extra test, run only inside the verifier.

• 

  Every step the agent took, recorded in a hash-chained log.

• 

  An attempt to read a secrets file, recorded as an attempt, not a leak.

• 

  The dashboard: an agent saying "all tests passed" is not proof.

• 

  Three seeded scenarios and the most recent runs.

• 

  The benchmark page shows every ratio with its counts.

• 

  Measured results, including the one miss.

Inspiration

We've all seen an AI coding agent finish a task and say "done, all tests passed." Most of the time we just believe it. But the agent is grading its own homework. It can be wrong, it can skip the hard test, or it can just say it ran something it never ran. We wanted a way to check the claim instead of trusting it, the same way you'd want a second reviewer on a pull request.

What it does

ProofLayer runs an AI coding agent on a small, bounded task and then checks its work independently.

• The agent works in a policy-scoped sandbox and can only use a small set of typed actions (read a file, write an allowed file, run a pinned check). It never gets a raw shell.
• Every action, tool result and the agent's final claim goes into an append-only, hash-chained event log.
• When the agent is done, a completely separate verifier (a Docker container with no network and no access to the agent) rebuilds the workspace and re-runs the pinned test suite.
• The verdict (VERIFIED, FAILED or INCONCLUSIVE) comes only from those measurements. The agent's claim is marked SUPPORTED, CONTRADICTED or UNVERIFIED.
• You can also "challenge" a claim: the model proposes one extra test, we check it for unsafe code, and only the verifier runs it.
• Each run gets a certificate with hashes of all the evidence, plus a copy of its digest in a separate ledger, so anyone can re-check it offline and edits get caught.

The demo has three scenarios: an honest fix, an agent that claims tests passed when they didn't, and an agent that gets nudged into reading a secrets file.

How we built it

• Model: NVIDIA Nemotron (nvidia/nemotron-3-super-120b-a12b) through Nebius Token Factory, using its OpenAI-compatible API with JSON output. Nemotron drives the agent in live mode and writes the proof-challenge tests.
• Backend: Python, FastAPI, Pydantic, SQLAlchemy, Alembic and SQLite. The agent loop is bounded by steps, tokens, calls and time.
• Verifier: a pinned Docker image that runs with no network, a read-only filesystem, no Linux capabilities, a non-root user and tight resource limits.
• Evidence: a SHA-256 hash chain over every event, a certificate manifest, and a detached ledger.
• Frontend: Next.js, React, TypeScript and Tailwind, with an event trace, a claim inspector, a certificate view and a challenges tab.
• Testing: 405 backend tests, 59 frontend tests, CI on GitHub Actions, and an 18-case benchmark that runs the whole pipeline three times.
• Hosting: the API on Render and the web app on Vercel, as a read-only replay of real recorded runs.

Challenges we ran into

• Sandboxing on a Mac. We installed NVIDIA OpenShell and got its gateway running, but Docker Desktop's Linux kernel doesn't have Landlock, so OpenShell (correctly) refused to start any sandbox. We couldn't fake that, so the agent runs in our mock sandbox and every certificate says so.
• Nemotron quirks. Its reasoning tokens count toward the reply limit, so at first one reply got cut off halfway through the JSON. It also sometimes escaped code twice inside JSON. Both showed up as rejected challenges (never executed), and we fixed both with tests.
• Running model-written code safely. We check challenge tests statically, but we don't treat that as the security boundary. The network-less verifier container is.
• Being honest in the UI. It's really tempting to show a nice green number. We made "INCONCLUSIVE" a normal outcome and refused to show any number we didn't actually measure.

Accomplishments that we're proud of

• A real end-to-end pipeline: claim, independent check, certificate, challenge. Not a mockup.
• On our 18-case benchmark: 18/18 correct verdicts, 4/4 false claims caught, 85/85 tampering attempts detected, and 0/50 safe actions wrongly blocked. It flagged 5 of 6 policy violations, and we show the one it missed.
• Three live runs with Nemotron, each with a certificate you can re-verify offline from files in the repo.
• In one live run, Nemotron actually fixed the bug in the scenario where our scripted agent lies, so that run came out VERIFIED. The verdict follows the evidence, not the scenario label.

What we learned

What's next for ProofLayer

Built With

• alembic
• docker
• fastapi
• github-actions
• nebius-token-factory
• next.js
• nvidia-nemotron
• nvidia-openshell
• openai-api
• pydantic
• pytest
• python
• react
• render
• sqlalchemy
• sqlite
• tailwindcss
• typescript
• vercel
• vitest