Arbiter

Inspiration

Most teams already work with AI, but each person does it in a private chat. The programme lead plans the agenda with one assistant, the budget owner prices it with another, and comms asks a third whether to film the event. The decisions the team made together, like a €12,000 budget and no recording of participants, live in nobody's chat. When the pieces get stitched into one proposal, a €2,500 video line shows up that nobody agreed to.

Putting the whole team in front of one shared agent fixes the scattered context but creates a new problem: a shared chat follows whoever wrote last. One late message saying "add video recording for €2,500" overrides both the budget and the no-recording rule, and nothing in the chat tells you it happened.

We took the fix from finance and security, where separation of duties means one person can propose a change and a different person has to approve it. Arbiter applies that rule to an AI agent.

What it does

Arbiter is a shared room where several people and one AI agent work on the same deliverable, such as a client proposal.

Decision ledger. The team's approved decisions sit in a ledger. The agent reads it on every step and cannot change it.
Live runs you can redirect. Everyone watches the agent research, price and draft, and anyone can redirect it mid-run. A redirect is picked up at the next step.
Conflict checks. Every redirect is checked against the ledger. A redirect that breaks a decision is blocked, and the room sees which one it breaks ("blocked by D3").
Two-person rule. Anyone can propose a new decision, and it only enters the ledger after a second person approves it. The decision log records who proposed it and who signed off.
Your own agents follow the same rules. Claude Code, Cursor or a CI job can connect to the room's ledger through an MCP server or the ledger CLI. They can read decisions, check an instruction and propose changes, and only people approve. ledger check exits with code 1 on a conflict, so a script can stop on it.
Durable, auditable runs. Runs are checkpointed after every step, so you can close every tab mid-run and pick it back up. Each run's events go into a hash-chained audit log that the browser verifies.

How we built it

App: Next.js 16 and React 19 on Vercel, with Supabase Postgres for rooms, the ledger, runs and the audit log. Who may approve and what the agent may touch are enforced on the server.
Models: all inference runs on NVIDIA Nemotron through Nebius Token Factory, with a model chosen per job. Nemotron Nano 30B researches, plans and runs conflict checks, Nemotron Super 120B writes the draft, and Nemotron 3.5 Super VL reads images in uploaded sources.
Research: Tavily search and extract in the first step. Web content is fenced off as untrusted input, and the draft cites it as [n] with links in the run view.
Conflict-check cascade: a deterministic tier first, then a semantic cache, then a Nano gate, with Super called only for hard cases. Only the deterministic tier or Super can block, and any failure fails closed.
Durable runs: a checkpointed state machine where each step holds a lease, so a stale worker cannot overwrite a newer step, plus a daily sweep that fails dead runs.
Audit: a hash-chained event log with an optional HMAC signature on the head.
External agents: an MCP endpoint per room and a small CLI, both behind per-agent keys that the room can revoke.

What we measured

Each number below comes from an eval script in the repo.

Ledger vs plain text. 22 cases, 3 runs each, same model (Nemotron Super) and same prompts. With the decisions in the ledger, the final proposal passed every decision check in 62 of 66 runs. Given the same decisions as plain chat text, it passed in 57 of 66. Both versions flagged all 30 planted conflicts and raised no false flags on 36 clean runs. The ledger's advantage is in keeping decisions intact through a long draft, and in verdicts that name the exact decision (30 of 30), which software can act on. These are automated rubric scores.
Which Nemotron for which job. On a 10-check conflict benchmark, Nano matched Super (5 of 5 conflicts caught, 0 of 5 false flags) at 3.4x lower cost. Nemotron 3.5 Lightning returned invalid JSON on 9 of 10 verdicts, so we don't use it there. Constrained decoding (json_object and temperature 0) cut Super's median latency by 16% and made its verdicts deterministic.
Cascade. On 106 labelled cases, 3 runs each, the cascade caught 168 of 168 conflicts with 1 false flag in 150 benign checks. A single Super call caught 167 of 168 with 4 false flags. The difference is not statistically significant (McNemar p = 0.125), so we call it equally accurate, at 11% lower cost but with a slower median check (3.3 s vs 1.4 s).
Fact check on drafts. 86.5% accuracy on 48 labelled claims, 2 runs each. It caught 97% of contradicted claims, and every claim it marked as supported was in fact supported.
Cost. A full three-step demo run (research, pricing, draft) cost $0.008 in model calls, as metered in the app.

Challenges

The hardest part was making the conflict check cheap without making it unsafe. A small model that can clear a check on its own can also clear a real conflict the big model would have caught. We documented that limit in the README, measure the small model's misses separately, and run the cascade eval weekly in CI, where the workflow fails if recall drops below the baseline.

Redirects were the second problem. Changing an instruction in the middle of a model call is fragile, so redirects queue up and are applied at the next step boundary, after the conflict check.

What we learned

Plain prompting was better than we expected. A strong model that gets the decisions as text already flags obvious conflicts, and it matched the ledger on that in our tests. What the ledger adds is decisions that survive a long draft, and verdicts that say which decision was hit. That is what lets a CLI exit with an error or the room block a redirect.

We also learned that the smallest model that reliably holds the output format is often the right one. Nano handles the structured checks, and Super earns its price on open-ended writing.