Scopewatch

Inspiration

Coding agents can read instructions hidden in files and turn them into tool actions. We wanted a review point before those actions have effects, with evidence a person can inspect.

What it does

Scopewatch routes agent tool calls through a gateway. Deterministic policy checks the requested action first and returns ALLOW, HOLD, or DENY. A policy denial is final. When reasoning evidence is available, an auditor can escalate an allowed action to HOLD. A failed audit also holds the action. A reviewer can grant an exact, single-use approval for a held action, and the controlled executor records the result. The dashboard distinguishes actions, decisions, reasoning provenance, approvals, and execution receipts.

The synthetic invoice scenario illustrates the idea: an injected instruction in an invoice asks an agent to change payment details. In the scripted demo, the policy permits the output write, while the auditor escalates it to HOLD based on the supplied reasoning evidence. This is a mock demonstration; the headline scenario has not yet been observed live on the Nebius profile.

How we built it

The gateway uses Python, FastAPI, Pydantic, and SQLite. The reviewer dashboard uses plain HTML, CSS, and JavaScript. Provider profiles make the agent and auditor models swappable. The Nebius Token Factory profile configures NVIDIA Nemotron 3.5 Lightning for inference. A historical held-out auditor-only run on 2026-10-03 is documented in the repository. It recorded 0 of 12 clean cases held per profile; these results do not establish current gateway performance or a live headline demo. Docker provides an isolated executor for supported commands. Synthetic invoice and coding workspaces make the local demo reproducible without credentials.

Challenges and what we learned

A model's reasoning is evidence, not proof of intent. Providers expose different fields, and Nebius did not expose a reasoning field in the documented profile. Scopewatch labels provider-exposed traces, agent-authored summaries, caller assertions, and missing evidence separately. We also learned to make an attempted audit fail closed, and to bind approval to one exact action rather than letting an auditor grant access. The gateway mediates only calls routed through its API; it cannot observe or block bypasses.

Accomplishments

We built the gateway, reviewer dashboard, single-use approval flow, synthetic scenarios, provider profiles, and an evaluation harness. The local demo runs offline with a mock provider. The repository includes testing instructions and a static replay of recorded gateway decisions using the mock provider.

What's next

Run the headline scenario live and select the agent and auditor profile pair. Complete real agent-runtime integration, measure the updated auditor on fresh cases, and improve evidence-chain operations and bounded networking.

Source and local testing instructions: https://github.com/dkritarth/scopewatch Static mock replay: https://dkritarth.github.io/scopewatch/demo.html License: MIT.

Built With

• docker
• fastapi
• javascript
• nebius
• nemotron
• nvidia
• openrouter
• playwright
• python
• sqlite