Vigil

Inspiration

On-call automation has a dirty secret: the moment you let a bot act on production, you hand it standing credentials, always-on keys that can roll back, restart, or delete anything, sitting in the agent's environment 24/7 whether it's mid-incident or fast asleep. Diagnosis was never the scary part; models already read logs well. Access is the scary part. A leaked token, a confused agent, or a single prompt injection turns "helpful bot" into "unbounded blast radius."

We wanted to prove a stronger model is possible: an autonomous on-call engineer that fixes real production incidents while holding zero standing credentials. Not fewer. Zero. Every destructive action has to be earned, one at a time, scoped, single-use, and expiring in seconds. That one constraint reshaped the entire architecture, and demonstrating it end-to-end is what Vigil is for.

What it does

Vigil runs a complete autonomous remediation loop against a live (mock) production service, a real HTTP service that serves real 500s and really recovers. Eight beats:

Break: payments-api starts failing under a live traffic loop; the measured error rate climbs.
Detect: a real threshold on that error rate trips and opens the incident. No human paged.
Context: Vigil pulls a governed read of the recent change set and pins the suspect: deploy #4821.
Buy the capability it lacks (Zero): the logs are a cryptic bespoke .vlog format Vigil can't parse. Instead of shipping a brittle hardcoded parser, it discovers and buys a log-parsing capability from Zero on demand, paying per call in USDC over x402.
Reproduce the failure (Akash): it ships the parsed evidence to a diagnostic worker deployed live on Akash, which reproduces the bug in a disposable sandbox and returns sandboxPassed=true, recommendedAction=rollback.
Earn a grant (Pomerium + gate): the remediation request transits Pomerium to a policy gate. Every check passes and a grant is minted: scoped to payments-api, single-use, 60-second TTL.
Recover: the rollback lands through the gated path, the error-rate curve really drops to baseline, the grant is consumed, and standing credentials return to 0.
Clamp: Vigil then overreaches and requests a mass-restart of 12 services. The gate denies it because blast radius is over the limit, policy tightens, and the audit strip shows every action, fully attributed.

The same UI drives both a live path (real services) and a deterministic scripted twin, switched by an env var, so the story is always identical and the demo never dies.

How we built it

The stack: a Next.js (App Router) frontend consuming the orchestrator over Server-Sent Events as full state snapshots; a vigil-agent orchestrator (Node/TypeScript) running the traffic loop, detection, and remediation; a purpose-built payments-api mock prod that serves real 500s and really recovers on rollback; and a policy gate that mints single-use, scoped, TTL-bound grant tokens. The three hardest problems (buying a missing capability, running risky compute safely, and gating destructive access) we solved with our sponsors, and each does real, load-bearing work.

Zero: the agent's "buy what you can't do" layer

Zero is what makes zero standing capability literally true, not just a slogan. When Vigil hits a wall, a .vlog format nothing in its toolbox understands, it doesn't fall back on a fragile in-house parser. It runs Zero's discovery-and-pay loop live: zero search "extract fields from raw text log line", then zero get, zero fetch, and zero review. Zero brokered us to the 402.com.tr AI Field Extractor (itself LLM-backed), and Vigil paid ~$0.03 per call in USDC over the x402 protocol on Base. Our wallet balance visibly ticked from 5.000 to 4.777997 USDC across testing, with a signed on-chain receipt for every single call.

This is genuinely profound for agent design. The agent accumulates no standing tools and no standing spend; it pays pennies for exactly the capability it needs, exactly when it needs it. Zero turned "least privilege" from a credentials idea into a capabilities idea, which is precisely the philosophy Vigil is built on. And Zero's marketplace did its job under honest testing: when our first pick (a generic log parser) couldn't handle our bespoke format, Zero's discovery surfaced one that could, and we swapped it with a one-line env change. A per-call, pay-as-you-go capability layer is exactly the missing primitive for safe autonomous agents, and Zero nails it.

Akash: disposable, verifiable compute for the riskiest step

Reproducing a production failure is exactly the work you want to run far from production, on hardware you can throw away. Akash made that trivial and real. Our diagnostic worker (da0t/vigil-diagnostic-worker) is deployed to a live lease on the overclock provider (na-us-west, 100% uptime) via the Akash Console, reachable at a public ingress URL. Vigil calls it live inside the loop: POST /diagnose returns sandboxPassed=true, a concrete root cause ("deploy #4821 changed stripe_adapter config handling, ERR_TIMEOUT_CFG"), recommendedAction=rollback, and all four diagnostic checks green.

Akash's decentralized compute marketplace gave us cheap, disposable, verifiable infrastructure for the one step we least wanted anywhere near our own prod, and it went from SDL upload to a live, bid-accepted lease with almost no friction. The sandbox is ephemeral by design: reproduce, report, tear down, no residue. For an agent that's supposed to leave zero footprint, running the dangerous step on Akash instead of on privileged infra isn't a convenience; it's core to the security story. Akash's decentralized compute marketplace gave us cheap, disposable, verifiable infrastructure for the one step we least wanted anywhere near our own prod, and it went from SDL upload to a live, bid-accepted lease with almost no friction. The sandbox is ephemeral by design: reproduce, report, tear down, no residue. For an agent that's supposed to leave zero footprint, running the dangerous step on Akash instead of on privileged infra isn't a convenience; it's core to the security story.

Pomerium: the only door to production's destructive routes

Pomerium is the load-bearing wall of Vigil's security model. We put it directly in front of payments-api so that the only two destructive routes that exist, /rollback and /restart, are reachable exclusively through Pomerium. Every other production path (/logs, /metrics, /deploys, /admin, /) has no route, so Pomerium answers 404 at the proxy and the request never even reaches prod. We verified this against Pomerium Core 0.33.0: routed calls return the backend's response; every blocked path returns Pomerium's own 404, proven right down to the response bodies.

Pomerium gave us identity-aware, policy-driven route enforcement as the outer layer of a real defense-in-depth model, and the gate re-verifies the single-use grant one layer deeper. So a destructive action has to satisfy both Pomerium's routing policy and a live, unconsumed, in-scope grant to land. Two independent gates that must both agree, exactly the way production access control is supposed to work. Pomerium made "the agent physically cannot touch anything but the two actions it's allowed to attempt" a verifiable fact, not a hope.

The gate and the fallback design

The gate mints tokens (vg_…) that are single-use (a consumed flag flips on first verify), 60-second TTL, and scope-checked on both action and service; reuse, expiry, or scope mismatch all fail closed, and standing() reads 0 at rest. Every integration also has a local mimic selected by env var (unset = fallback, set = real sponsor service), which let four of us merge four branches under a hard deadline and keep a working demo at every single stage. An optional LLM step writes the one-sentence root-cause hypothesis; the loop runs identically without it.

Challenges we ran into

Merging four parallel branches against a submission clock. We solved it with a frozen shared wire contract plus an env-var fallback for every integration, so main was demo-ready after every merge, with no big-bang integration at the end.
Making single-use actually single-use across a trust boundary. The destructive service can't just trust the agent's word, so it independently re-verifies each grant with the gate. A token can transit Pomerium and still be rejected downstream if it's spent, expired, or out of scope.
Wiring Zero's real CLI + USDC wallet (x402) flow into an agent cleanly, and being honest when the first capability we bought didn't fit our format, then letting Zero's discovery route us to one that did.
Deploying to Akash from ARM Macs (the --platform linux/amd64 flag is mandatory, not optional) and driving it to a live, verified lease with a real ingress URL.
Keeping the narrative honest. We shipped a public "what's real vs simulated" table so judges know exactly where the real 500s, real detection, real payments, and real recovery end.

Accomplishments that we're proud of

Zero standing credentials, provably. Every destructive action is a scoped, single-use, expiring grant, and the gate reports 0 standing credentials at rest.
A recovery curve that's real. The error rate drops because the rollback really flips the service. Nothing animated.
A real denial, on stage. The gate refuses the 12-service mass-restart because blast radius exceeds the limit. The "no" is the whole point.
Three live sponsor integrations doing load-bearing work, not logo decoration: a paid Zero capability call with an on-chain receipt, a diagnostic worker genuinely running on an Akash lease, and destructive routes that physically only exist behind Pomerium.
Signed, per-call payment receipts for every capability the agent used, auditable spend to match the auditable access.

What we learned

Least privilege for agents is a design pattern, not a config flag: just-in-time, scoped, single-use grants collapse an autonomous agent's blast radius to exactly one approved action. Zero taught us to extend that same principle from credentials to capabilities: an agent that buys tools per-call over x402 accumulates no standing power of any kind, financial or functional. Akash showed us that the safest place to run a risky reproduction is disposable, decentralized compute you don't own. And Pomerium reminded us that real security is layered: the outer route policy and the inner grant check should be independent, and both should have to say yes.

What's next for Vigil

Real connectors: Kubernetes, cloud deploys, and real observability behind the same Pomerium + gate path.
A richer policy language and human-in-the-loop approvals for higher-blast-radius actions.
A full Zero capability marketplace so the agent can buy any diagnostic tool per-incident, priced per call.
Tightening Pomerium itself with a PPL rule that denies any request missing a valid grant header, enforcement at the proxy on top of the gate.
Auto-generated postmortems built straight from the attributed audit trail. Tip: Paste images into Claude Code using control+v (not cmd+v!)