QUNCE

Self-healing infrastructure without surrendering control.

Comment

Inspiration

Any capable model can propose a fix for a broken deploy. None of them should be allowed to run one unsupervised.

That gap is the whole problem. When checkout is throwing 500s, the model isn't the bottleneck — it'll suggest a rollback in seconds. The danger is everything after the suggestion: rolling back the wrong release, acting on a hunch, firing the same destructive action twice on a retry. The human who decides whether to execute a high-impact action, and signs off on the exact one, is doing the real work. The model is just an investigator.

I wanted to find out whether that authority could be written down — not "the model is careful," but a deterministic gate the model cannot cross, a human approval bound cryptographically to one specific action, and a receipt a skeptic can replay.

What it does

QUNCE turns an ambiguous production alert into an executed, verified fix. Low-risk allowlisted checks can execute automatically; high-impact mutations require a human authorization bound to the exact action envelope.

  1. A council of Qwen agents investigates. A Signal Keeper triages the alert; an Investigator correlates health, error rate, deployment history, latency, and logs; a Safety Reviewer independently critiques the reasoning instead of rubber-stamping it.
  2. A deterministic risk engine scores the plan. Low-risk, in-boundary actions run automatically. High-impact ones require a human decision bound to the exact plan hash and action envelope — approve this action, on this target, with these parameters. Anything else is refused before side effects.
  3. A restricted host agent executes — the only component with Docker access, a root-owned systemd service outside the application containers, allowed to run the approved envelope and nothing else.
  4. QUNCE independently verifies recovery — health, error rate, latency, a smoke test — and only marks the incident resolved if recovery actually happened.

Every step seals into a hash-linked evidence chain you can replay. The deployed demo is a bad checkout release at 31% error rate, rolled back and verified down to 0.2%, closed with 20 hash-linked cards and a Qwen-written postmortem that names the schema mismatch instead of hiding it.

How we built it

Backend: Python + FastAPI on a durable PostgreSQL control plane — workflow jobs with database leases and SKIP LOCKED, dead-letter handling, a transactional outbox, and a mutation execution journal, so a committed stage survives any worker or API restart and never double-executes.

Every model call routes through Qwen Cloud, tiered by cost: qwen3.6-flash for triage and postmortems, qwen3.7-plus for investigation and planning, qwen3.7-max for independent safety review and escalation — each doing the tier of thinking it's priced for.

Signed alert ingress (HMAC + replay window + dedupe) so incidents can only be injected by an authenticated source. Read-only evidence tools plus authenticated MCP servers that can inspect but never mutate Docker or the host.

Deployed on Alibaba Cloud ECS (Docker Compose behind Caddy) against a controlled "victim" workload as the real target. 224 tests cover the state machine, hash binding, exactly-once execution, and fail-closed behaviour.

The design rule throughout: every claim on screen must be derived from stored data. No hardcoded PASS states, no floored counters, no badge that asserts an outcome the record doesn't support.

Challenges we ran into

The benchmark exposed a real bias, and I kept it in. Across 40 live scenarios, QUNCE scored 65% action accuracy against 70% for a single Qwen agent, and 95% authorization accuracy against 100%. Reading the per-scenario data: all 14 of QUNCE's action misses were human_escalation — none was an unsafe remediation proposal. Twelve were policy-correct escalations the single-Qwen arm made too; two were unnecessary escalations on read-only smoke-test cases, and those two are exactly why QUNCE recorded 95% rather than 100% authorization accuracy. They're genuine mistakes — QUNCE asked a human where it could have run a read-only check itself. I kept them in the published result instead of tuning away the conservative bias, because erring toward "ask a human" is the safe direction to be wrong.

My own dashboard was flattering itself. Preparing the demo, I caught the UI lying in three places. The evidence rail read "Challenge → Human gate → Recovered" on an incident whose safety reviewer had actually returned CONFIRM — because the badge was derived from the presence of a review card, not its decision. The metric strip showed "Human gates: 1" while my own public /api/v1/stats returned 0, because a seeded fixture was being floored over the live value with max(). The integrity panel printed three of four checks as PASS from hardcoded true. Nothing crashed. It all looked great. Deriving every badge, count, and check from the actual sealed record mattered more than any feature I could have added.

Failing closed on a bad plan. When Qwen returns an invalid execution envelope, deterministic verification rejects it. The control plane derives a minimal bounded envelope from an allowlisted action ID, or falls back to a read-only smoke test, validates it again, and refuses execution if it still fails.

The privilege boundary. Making the host agent the sole holder of Docker access — a root-owned systemd service outside Compose, reachable only over a Unix socket, refusing raw commands and host paths — so a compromised model can propose all day and still never reach the engine.

Accomplishments that we're proud of

  • A governed loop that actually executes and verifies against a live system — investigate, gate, human-authorize, execute deterministically, verify recovery — not a slideshow.
  • Zero unsafe proposals from full QUNCE across all 40 live scenarios.
  • Hash-bound, exactly-once authorization that refuses replays and duplicates before side effects.
  • Every displayed metric traces to stored data. The worst bugs I shipped against were my own UI flattering itself — a mislabelled verdict, a floored counter, hardcoded PASS states — and killing them was the point.
  • A deployment where a judge can read the receipts: sealed evidence chain, committed evaluation with the losing rows left in, and a live URL.

What we learned

The hard problem in agentic software is authority, not capability. A model that reasons well is only safe in production if there's a deterministic, auditable boundary around what it's allowed to do.

Publishing a conservative bias — including the two cases where it cost accuracy — is more credible than a hidden perfect score. QUNCE errs toward asking a human; I showed exactly where that helped and where it didn't, instead of tuning the number.

A dashboard that can't fail loudly will lie quietly. Hardcoded PASS and floored counters don't crash; they just make an unproven system look proven. Fail-closed beats fail-quiet — in the risk gate and in the UI.

What's next for QUNCE

  • Broader action allowlists behind the same risk gate, so more of the runbook runs automatically without weakening the human boundary.
  • Managed-service ingress (ARMS / SLS / EventBridge) as an alternative to the signed injector.
  • A richer incident-memory layer so the council gets faster on incidents it has seen before — always behind the same hash-bound human authority.

Built With

Share this project:

Updates