Arbiter

Arbiter is an AI agent that hires other AI agents to do real work, and only pays them once an independent evaluator has actually verified the job was done.

Comment

Inspiration

I have been using AI agents for small tasks myself, cleaning up data, drafting quick things, and I kept noticing the same moment of discomfort every single time, the instant I paid, I had already given up any leverage to make sure the work was actually right. That stuck with me enough that I went looking into whether this was a known problem rather than just something in my head, and it turns out it is, research on agent payment protocols like x402 has found that payment success and service delivery are only weakly coupled, meaning the money moving and the work actually happening are two separate promises that nothing forces to line up. I also came across a paper looking at the leading on-chain reputation standard for agents, ERC-8004, and the numbers surprised me, only three to fifteen percent of registered agents were running a real service, and up to ninety point six percent of on-chain reviews turned out to be Sybil coordinated fakes. That was the moment this stopped being a vague feeling for me and became something I wanted to actually build a fix for.

What it does

Arbiter is an AI agent that hires other AI agents to do real work, and only pays them once an independent evaluator has actually verified the job was done. A person describes what they need in plain chat, translate a listing, clean a spreadsheet, caption a clip, and Arbiter hires a worker agent, funds an escrow contract instead of paying the worker directly, and waits for verification before any money moves. Verification runs in two stages, a deterministic structural check first, then a rubric locked panel of two different language models voting independently, so a single model cannot rubber stamp its own mistake. If the work passes, the contract releases payment automatically. If it fails, the contract refunds automatically. And above a spend threshold the agent does not get to decide on its own at all, it pauses and waits for a human to approve, with that threshold enforced on chain rather than sitting quietly inside a prompt.

I proved all three of these behaviours live on Base Sepolia rather than just describing them. A translation job that came back correct settled and released payment automatically. A job where I deliberately submitted the untranslated source text as the translation was rejected by the judge panel and refunded automatically. And a job priced above the approval threshold reverted on chain with a PolicyGuard error until a human actually signed off, at which point it funded successfully.

How I built it

The escrow itself is a fork of ERC-8183, the emerging agentic commerce standard, and the one genuinely new contract I wrote on top of it is PolicyGuard, which enforces a monthly spend cap, a worker allowlist, and a human approval requirement above a threshold, all in code rather than in a system prompt an agent could talk itself out of following. The principal agent runs on Groq, using Llama 3.3 as its own reasoning and tool calling model, together with Coinbase AgentKit for wallet and payment handling, and it hires workers, funds escrow, polls the evaluator, and settles or refunds, with a harness tracer attached that returns a structured trace on every chat turn proving each part of the agent loop actually fired rather than just claiming to. The worker services sit behind a real x402 paywall and do real work, translation through SEA-LION, video captioning, and spreadsheet cleanup. The evaluator is a small FastAPI service running the structural check and then the cross model judge panel, Llama 3.3 and gpt-oss through Groq, with self consistency voting and a split verdict escalating straight to a human. The frontend has two modes, a plain chat interface for the person who just wants something done, and a full operator console showing PolicyGuard state, the job ledger, and the approval queue for whoever is actually managing spend policy.

Challenges I ran into

The hardest part was not writing tests, it was checking whether the code those tests covered was actually reachable from a real request rather than only exercised in isolation. Running that kind of adversarial self audit against PolicyGuard turned up a real vulnerability, the contract recorded that a job had been approved but not what budget it had been approved for, so a human signing off on a small spend could have that approval silently reused later if the job's budget was raised past what they actually agreed to. I fixed it by binding approval to the exact approved amount rather than just a yes or no flag, and I wrote both an attack test and an exhaustive symbolic proof against that exact binding so it cannot regress quietly. Partway through the build I also lost the password to my original deploy key, and since that key held the escrow's admin role with no recovery path, I had no choice but to redeploy fresh from a new key and rewire every reference to the new addresses, which was a genuinely stressful day but also forced me to make the deploy process itself repeatable rather than a one off.

Accomplishments that I am proud of

I am proud that almost nothing in this build is simulated. All three demo moments, the automatic release, the automatic refund, and the human approval gate, were run for real against Base Sepolia, with real transaction hashes, not scripted for a video. PolicyGuard is backed by twenty six Foundry unit tests, two fuzzed invariant tests run over one hundred twenty eight thousand calls each, four Halmos symbolic proofs that hold exhaustively across the entire input space rather than a sample and run in continuous integration on every push, and a Slither pass with zero medium or higher findings. Across the whole stack there are one hundred eight automated tests, and almost all of them run against real APIs and real chain state rather than mocks.

What I learned

I learned that trust in agent to agent payments is not really a reputation problem, it is a verification problem, and reputation systems keep getting built because verification is harder and less glamorous to implement. I also learned, the hard way, that testing in isolation is not the same thing as testing what an attacker can actually reach, and that the only way I found my one real vulnerability was by deliberately trying to break my own assumptions rather than trusting that high test coverage meant the system was safe.

What's next for Arbiter

I want to widen the worker pool beyond the three tasks I support today, translation, captioning, and spreadsheet cleanup, and let more Southeast Asian small businesses actually plug real jobs into this instead of a demo script. I also want to close a gap I found while reading around my own work, my evaluator currently reasons closer to a self reported or witnessed tier of evidence than to a cryptographically provable one, and there is recent research pointing at admissibility frameworks that grade verdicts by the evidence class behind them rather than just by whether judges agree, which is the honest next step for making verification itself harder to game.

Built With

Share this project:

Updates

Submission history