About the project
Inspiration
The idea came from a repeated annoyance. Every time I asked an AI agent to do a coding task it claimed it succeeded, and I had no cheap way to check. Reading the diff myself was just doing the work twice. When I did bother to check, the failures were never exotic. The agent edited the test suite to make it pass. It ran one command twice and called it done. The problem wasn't capability. Nothing between the agent and me was checking anything.
So NorthStack started as one question: what does it take to run an autonomous AI agent where "verified" actually means something? I wasn't trying to build a smarter agent. I wanted a control plane that plans, budgets, and checks the work, and keeps an honest record of all of it.
How I built it
NorthStack is a provider-neutral control plane that runs agent work as task cells. A cell is a small unit of work with an immutable goal contract, a budget, and a row in a dependency graph. A few pieces took real design work.
The contract comes first. Before a cell runs, an acceptance analysis proposes hard-checkable criteria, and a falsifier tries to break the contract before it's paid for. The model plans the decomposition, then deterministic code hardens it: ids get rewritten, dependency waves recomputed, budgets clamped. If it can't be hardened it's thrown out for the canonical single-cell plan. The model proposes, the solver disposes.
Verification happens in two layers. Hard gates run real workspace operations, like commands, file assertions, and JSON Schema checks, and there is no way to waive them. On top of that sits a blinded soft-review panel. Reviewers see the actual artifact content but they structurally cannot see who the executor was, and every way a reviewer can fail resolves to "disagreement" instead of a pass. Since any panel can wrongfully accept work, my accept criterion requires surviving both layers:
$$P(\text{accept}) = P_{\text{hard}} \cdot P_{\text{soft}}$$
Rejection needs only one layer, acceptance needs both, so the joint probability stays small by construction.
The ledger keeps the record. Every event goes into SQLite with a SHA-256 hash chain, so a run can be replayed and audited later without trusting its own summary. A crashed run is retained as a failure and never dropped, because a dropped failure is exactly the kind of record that quietly inflates a success rate.
Finally, I wanted to prove the thesis instead of asserting it. The claim standard is that the machinery earns its complexity only if held-out experiments show better verified quality at equal resource use, or equal quality for fewer tokens, calls, or dollars. That's why there is a benchmark runner with four preregistered configurations, paired bootstrap intervals over 2000 resamples, and scoring done post-hoc by hidden checks that the system under test never sees. A live run, decomposed into four cells with routing between roles, cost $0.0385 end to end, spent 22,171 tokens in and 11,159 out, and came back verified.
Challenges
The hardest problem was authority sprawl. The surest way to make a budget meaningless is to let three subsystems each enforce it "mostly". I ended up giving every dangerous power exactly one owner. One class owns the retry cap and reroute, one owns spend, and one is the sole constructor of a run's outcome. Getting there took a few rewrites after I caught two components both deciding whether to retry a failed call, each with slightly different caps.
The second challenge was evaluating a system designed to be honest. If my scoring rubric is soft, a fluent but wrong run can pass every check. I watched that happen once and re-verified everything with cold, executable gates after that. The soft rubric still catches what hard gates can't, but it never gets the final word alone.
The third was deployment, making the cloud build reproducible and keeping credentials server-side only. Nothing but API keys lives outside local configuration, and the demo runs entirely against the deployed instance.
What I learned is that the boring parts, like budgets, ledgers, gates, and one authority per decision, are where agent reliability actually lives. Hooking up a language model was the easiest part of the system.
Log in or sign up for Devpost to join the conversation.