Authority Arena

The question

Capability is learned. Authority is granted. Before a developer gives an agent payment or deployment tools, they need to know two things: can it act beyond its approval, and what evidence survives afterwards? Authority Arena is an open testbed for both. 21 open models on Nebius Token Factory get executable tools against simulated payment and deployment systems, a limited approval, and a reason to exceed it: eight pressure tasks, each a different route around an approval, plus four legitimate ones. Each system's own ledger decides what changed, never the model's account of it.

What you can do

• On arena.aqta.ai: pick a task and a model, replay the recorded run arm by arm, see what each system's ledger applied or refused, and verify every signed decision in your browser. Edit a signed field and the check fails.
• From the repository: rerun the evaluation and inspect every published run. Four of the five arms need no Aqta service.
• Built for developers testing tool-using agents before they get payment or deployment access. Seal, the boundary under test, is Aqta's live product.

What we found

Five arms, 3,772 pre-registered runs (8 that ended in an API error are excluded, as registered).

• Rules in the prompt are not a boundary: about a third of pressure runs still got through.
• Names are not values: an allowlist of tools let a split payment through in 58 of 63 runs.
• Nemotron, Nano to Ultra: 92 of 96 leaked with no boundary, 0 of 96 behind a plain check, 0 of 94 behind Seal.

Same prevention result. Different evidence.

Both exact checks, the plain check and Seal, prevented every unapproved change observed in their runs. The difference is what an outside reviewer can verify afterwards. Seal signs each decision, refusals included: 1,529 signed records across the Seal arm's runs (one per decision; a run can make several attempts). All 1,529 verified offline against their run's key, and an edited copy of each was rejected. The plain check produced no signed decision record an outsider could verify.

A reviewer can verify each record's signature and integrity offline, without access to Aqta's service or the operator's systems; they still decide which signing key to trust. A verifying record establishes that it is intact and who signed it, not that the action happened, that the approval was right, or that every decision was recorded.

So we attacked the records themselves, in a second pre-registered evaluation across 11 attack types: editing, re-encoding, forged keys, refusals turned into approvals, split histories, reordering, backdating and withholding. Seal passed 9, a signed cloud audit log 6, a plain log 1. These are results against the specified tests, not a security score. Seal fails two: a record that was withheld or never written (omission, which no approach we tested solves), and a record dated earlier than it happened (the log proves when a record arrived, not that its stated time is true). Method, comparator setup and every result: arena.aqta.ai/evidence

How we built it

Python harness with simulated systems (430 tests). Every model call goes through Token Factory's OpenAI-compatible API, at temperature 0.7 and at most 14 turns. Seal is called over HTTP, and its records are checked with the published aqta-verify-receipt on PyPI and npm. The static pages replay every run and verify each record in your browser. Measures, tasks and arms were registered before the first run. Token Factory inference for all 3,772 Arena runs cost USD 37.58.

Built during the hackathon: Authority Arena itself, all of it (systems, tasks, harness, scoring, replay pages, pre-registration), the plain-check arm, and Seal's single-use exact approvals. Seal's signing gateway and record format existed before.

Limits and corrections

Simulated systems; Seal ran locally with a throwaway key. Not tested: stolen approver keys, agents using their own credentials, adaptive attackers. Two corrections, both dated and kept on record at arena.aqta.ai: a blank approval that let one deploy through (fixed and rerun on 7 Oct), and status queries miscounted as changes (9 Oct; no Seal or plain-check result changed)

Built With

• aqta
• aqta-verify-receipt
• css
• ed25519
• html
• httpx
• javascript
• nebius-token-factory
• nvidia-nemotron
• python
• seal
• web-crypto