Ouroboros: Self-Healing SaaS QA Loop

A self-healing SaaS agent swarm that finds bugs, fixes them, verifies the fix, and gets measurably better each cycle, all governed so nothing ships without the right approval.

Comment

Inspiration

AI coding agents can ship a feature in minutes, but almost nobody tests what gets shipped. Most AI-built apps go live with zero QA coverage, and the first "test" is a real user hitting a broken flow. We wanted to close that loop: an agent that finds its own bugs, fixes them, checks its own work against the live app, and is honest about which parts of that pipeline are real versus simulated because a self-healing loop you can't verify isn't trustworthy enough to hand real code to.

What it does

Ouroboros is a governed, self-verifying QA loop for a seeded SaaS app with four real, planted bugs (an off-by-one in pagination, a null-pointer crash, a race condition, and a missing-validation hole):

  1. Triage: a Triage Agent classifies each bug's severity and whether it touches protected code paths.
  2. Fix: a Fix Agent queries Senso's real knowledge base (our coding standards + a past incident postmortem, uploaded to a live Senso workspace) for grounding, checks Actian VectorAI DB for similar past fix attempts from earlier in the same run, then drafts a patch. When a real model key is present, the model's own diff is parsed and applied directly via the system patch binary, not rewritten.
  3. Govern: every patch is checked against a real, enforced policy. Anything touching app/core/ or app/db/ blocks the loop until a human clicks Approve or Reject in the dashboard. Every decision is logged.
  4. Apply & Verify: approved patches are written to the live app, which restarts, and a Verify Agent fires real HTTP requests at the restarted process to confirm the bug is actually gone, not a simulated check. A failure sends the bug back to Fix with the failure reason, up to two retries.
  5. Evolve: every fix attempt's outcome is embedded and stored in Actian, so later bugs in the same run are drafted with real memory of what worked before. The dashboard's confidence metric is the Fix Agent's own rolling first-try verification success rate. In one measured run, all 4 bugs resolved on the first try, confidence climbed from 0.553 to 0.681 across the run (a +0.128 trend), averaging 0.642.

We also ran Replay QA's actual crawler against the live app (tunneled out via cloudflared) mid-build and watched it explore the app for real.

How we built it

Python/FastAPI backend, SQLite for state, React + TypeScript + Vite dashboard over REST and WebSocket. Every sponsor tool sits behind a small adapter interface (backend/adapters/) with an identical real-vs-mock signature, so a key drops in with zero refactor. We were strict with ourselves about only calling something "real" once we'd verified it end to end, not just imported a client library.

  • Senso: their public docs only documented the /org/search endpoint, not ingestion, so we uploaded our grounding docs through their dashboard and verified the search call directly with curl before wiring it. Every one of the 4 Fix Agent runs in our test came back grounded in a real citation, e.g. coding_standards.md, postmortem_race_condition.md.
  • Actian VectorAI DB: ran their Community Edition in Docker and hand-probed the REST API with curl (PUT /collections, PUT .../points, POST .../points/search) until we confirmed it was Qdrant-compatible. No official quickstart repo was publicly reachable. It backs a genuine fix-memory: in the same run, 3 of 4 patches recalled a real, embedded, vector-similar outcome from an earlier bug fixed moments before.
  • Guild: we deliberately did not fake this one. Their product is agent hosting via a TypeScript SDK with browser-based login and chat UI invocation. There's no documented synchronous endpoint a Python backend can call for an instant approve/deny decision, which is what a live-blocking governance gate needs. So our governance policy is real, logged, and actually blocks the loop. It just isn't calling Guild's platform, and we say so.
  • Band: adapter is real in-memory pub/sub persisted to SQLite. The dashboard's handoff timeline is driven by it, with 18 real handoff messages logged across Triage → Fix → Gate → Human → Verify in our test run. We couldn't find a public self-serve API to wire their actual message bus in the time we had.
  • Pioneer: calls Anthropic's API directly as a stand-in when a key is present, with the exact signature Pioneer's own OpenAI-compatible endpoint expects, so swapping in the real one is a one-line change, not a rewrite.

Challenges we ran into

  • A real race condition in our own verify step: checking the app's health immediately after a patch could hit the still-running old process before uvicorn --reload actually restarted it, giving a false "still broken" result. Fixed with a process-start-time marker the verify step waits to see change.
  • Two bugs (BUG-2, BUG-4) land in the same file. Once we made bug-processing concurrent so an auto-approved fix isn't stuck behind one awaiting human approval, we had to add a per-file lock so their concurrent patches couldn't clobber each other.
  • Sponsor API docs were frequently incomplete or gated behind private Discords or hackathon-only channels. Actian's ingest endpoint and Guild's actual invocation model both took direct experimentation, or reading their docs closely enough to discover the product shape doesn't fit our architecture, rather than following a quickstart.

Accomplishments that we're proud of

  • All 4 planted bugs go triaged → fixed → governed → applied → verified → resolved end to end, first try, with zero retries needed. The only manual step is the approval click on the 2 of 4 bugs that touch protected paths (the other 2 auto-approve correctly).
  • Three sponsor tools are genuinely called over the network every run: Senso, Actian, and Pioneer (when keyed), verified by hand before being wired, not assumed to work from documentation alone.
  • A confidence metric that's an honest signal (rolling real verification success rate, +0.128 trend across a clean run) instead of a number designed to look good on a chart.

What we learned

Being upfront about what's mocked is more convincing than pretending everything's wired because a technical judge can tell the difference between "we imported your SDK" and "we verified this call returns 200 and grounds a real patch." The harder, more interesting engineering problem was never getting an agent to draft a plausible-looking fix. It was building the verification and governance steps that make it safe to trust that fix without a human reading every diff.

What's next for Ouroboros

  • Wire Guild the way their platform actually works: a deployed agent judges can open in a chat session, rather than forcing it into a synchronous shape it doesn't support.
  • Get Replay QA's programmatic crawl API wired instead of the manual UI run we used so exploration re-triggers automatically on every verify pass.
  • Get Band credentials once self-serve access exists, and persist Actian's fix-memory across process restarts instead of per-session.

Built With

Share this project:

Updates

Submission history