Inspiration
Personal AI assistants can now act for you: send email, reply to threads, book things. Today you choose between approving every single action, or giving the assistant free rein. Approve everything, and by the twentieth "yes" you stop reading, so the one that matters slips through. Give free rein, and one day it promises a client something you never agreed to. That is approval fatigue, and it is the reason most people keep their assistant on a short leash.
What it does
Clearance is a permission and approval layer between an assistant and the tools it acts with. Give permission once. Your assistant knows when to ask again.
- You approve a standing permission one time, with clear boundaries: what it may do, who it may contact, what it may share, what it must never share, how many times, and until when.
- Inside that permission the assistant works quietly.
- It stops only at a meaningful exception (a new recipient, a new commitment on your behalf, information you did not agree to share), with one card that says why.
- Every permission is visible and revocable. Repeated approvals can suggest widening a permission, but never widen it silently.
In the demo, an assistant follows up on a pilot project: one permission card, four replies sent without asking, and one draft held because it promises a Monday go-live. 2 approval cards instead of 5. Try it at the Hugging Face link below (a replay of Nemotron's recorded answers, with every decision made by Clearance's real code), or run it live locally from the repo.
How we built it
- Gateway: Python (FastAPI) with an MCP server (
clearance-mcp), so an assistant routes its actions through Clearance instead of acting directly. It also plugs in over plain HTTP. - Rules first: the permission is checked exactly: recipients, use limit, expiry. Anything outside it is a question, not a send.
- Judge: for drafts inside a permission, Nemotron 3 Super on Nebius Token Factory answers three yes/no questions: does it make a new commitment, does it share something the permission does not allow, is it off purpose? Strict JSON output, low reasoning effort, about 2 seconds per check.
- Thresholds: each question has a cost for missing it (commitment 20, sharing 50, off purpose 10, in units of one interruption), and the ask threshold is fitted once on a separate calibration split.
- Safety property: the judge can only turn a send into a question. Model confidence never grants permission.
Results
On 34 held-out test emails (16 routine, 18 that needed a person):
| Setup | Routine emails interrupted | Boundary misses |
|---|---|---|
| Ask before every email | 100% | 0% |
| Permission rules only | 0% | 78% |
| Clearance + Nemotron, fitted thresholds | 6% (1 of 16) | 0% (0 of 18) |
Challenges we ran into
- Open local judges. We also tried two open models as the judge, fine-tuned on about a thousand training emails we generated: CLM (projection heads on a frozen Qwen3-8B encoder) still asked about 13 of 16 routine emails, and Laya fine-tuned with RLCD stopped asking but let 3 to 5 of 14 real problems through. Nemotron was better on both counts, so it stays the judge.
- Catching our own leak. A pre-merge review found that an earlier training split had drifted toward the eval set and made Laya look as good as Nemotron. We rebuilt the split with tests that keep it away from the eval, retrained, and report the corrected numbers.
- Calibration. Taking the model's probabilities at face value interrupted on 31% of routine emails; fitting a per-judge threshold scale on the calibration split brought it to 6%.
What we learned
- Prompted models are not calibrated at the few-percent level; thresholds have to be fitted, and the fit has to happen on data the eval never sees.
- Nemotron 3 Super reasons before answering;
reasoning_effort: "low"keeps structured answers reliable within a token budget.
What's next
Email is the first surface. The same shape (one permission, quiet work inside it, a card only at the edge) fits calendar invites, messages, purchases and file sharing, and the repo already applies it to coding agents. Next up: a real mailbox connector (Gmail, Microsoft 365) and calendar permissions.
Log in or sign up for Devpost to join the conversation.