Codify

Learned from Usage
The Specialized AI Agent
Compound split
Showing Traces
Logo
Codify - Architecture Diagram
Routed to Our Specialist

Inspiration

We started with two frustrations that looked unrelated. The first was that least-privilege for agents is a policy nobody can actually write. You cannot know which hosts a task will reach or which directories it will touch until you have watched it run a few dozen times, so the scope you write on day one is either so wide that it grants nothing meaningful or so narrow that it breaks the work on the first unusual input. Everyone ends up running agents with full access and promising themselves they will tighten it later. The second frustration was that a single task gets asked fifty different ways. Five people who all want release notes from the same repository will write five different prompts, and they will get five differently structured documents back, different headings, different grouping, different level of detail, for what was, in substance, one job. Those two collapse into one observation: repetition is the signal for both problems. If you can tell that fifty phrasings are the same task, you can distil one brief out of them, and the same fifty runs have already told you exactly which hosts and which paths that task genuinely needed. One piece of evidence, two payoffs. The important part, and the reason we think Codify has a chance of being used rather than admired, is that the quality half is the adoption mechanism for the security half. Nobody installs middleware in order to be restricted. They install it because their team's output stops being inconsistent. Least privilege then arrives as a side effect of something people already wanted.

What it does

Codify sits between the user and the agent runtime and moves work through five stages: observe, promote, route, enforce, refine. In the observe stage every run is recorded but nothing is blocked, the prompt, the egress attempts, the filesystem reads and writes. Once a cluster of prompts is recognised as one recurring task, Codify promotes it into a specialist contract: a distilled brief assembled from how the task is really being asked, plus a scope derived from what those runs actually touched. From then on, incoming requests are routed by matching them semantically against existing contracts, and enforcement happens at the container boundary rather than in the prompt. Corrections that recur get folded back into the brief, which versions the contract. Enforcement runs across three channels, network, filesystem, and secrets, and they do not fail in the same direction, which is why we kept them separate rather than collapsing them into one permission list. Network is deny-by-default against an allowlist: too narrow and the task visibly breaks, too wide and data leaves quietly. Filesystem is the inverse in practice, because an over-wide write scope silently corrupts state that other tasks depend on while an over-narrow one fails loudly and harmlessly. Secrets are the strictest of the three: nothing is injected unless the contract names it, so the default is no capability at all. A promoted postmortem task, for instance, ends up with no egress whatsoever, the run reaches nothing. Two design decisions matter more than the rest. Routing fails open: when no contract matches a request, the run proceeds observed rather than blocked. Middleware that refuses unrecognised work gets uninstalled within a week, and failing open also means Codify's first day in a codebase is pure observation with no behaviour change. Scope, on the other hand, does not depend on recognition. Once a run is inside a specialist, an unrecognised follow-up prompt is still bound by that specialist's permissions, recognition decides routing, never enforcement. This is the property that makes the refusal table meaningful: an unapproved host stays unreachable, a write outside the contract's writable paths fails against a read-only mount, a prompt the specialist does not recognise still runs under its scope, and a contract past its token ceiling stops regardless of what is being asked. Multi-part requests are split, and each part is routed on its own under the contract that recognised that part, never under the union of two contracts' permissions.

How we built it

The whole system is TypeScript. Agent runs execute inside a Docker container against the Codex CLI, and model calls go out through Ark on two endpoints, one for the agent's own completions, one for the embeddings that routing and clustering depend on. Between the container and the network sits our egress broker, written deliberately without dependencies: it holds the per-contract allowlist, refuses anything not on it, and records every refusal as evidence rather than dropping it silently. Around all of it we built measurement tooling, because most of the claims we wanted to make about routing and scoping are only worth making if they are measured on a corpus rather than demonstrated once on stage.

Challenges we ran into

Our first clustering approach used Jaccard similarity over prompt tokens, and on a realistic corpus it would never have promoted anything at all, fifty genuine phrasings of one task share far less literal vocabulary than intuition suggests. That pushed us to embeddings. The embeddings then introduced their own failure: with single-linkage clustering, one bridging prompt is enough to chain two unrelated tasks into a single cluster, because A joins B and B joins C even when A and C have nothing to do with each other. The worst bug was subtler. We were reusing a shared Codex thread across runs, and the agent began answering from conversation memory instead of actually reading the repository. The runs looked successful, which meant the observed scope was being derived from work the agent had not really done, the observation layer was recording a fiction. Separately, our ab.chatgpt.com broker tests were being silently skipped because of a shebang line in the test file: the suite stayed green while the tests that mattered most never executed once. And the multi-task split path shipped without computing embeddings for the individual parts, so split requests fell straight through to unmatched even when perfectly good contracts existed for both halves.

Accomplishments that we're proud ofCodify

Promotion is correct on 12 out of 12 tasks across a corpus of 1,748 prompts. On WorkBench we measure a 10.6% misroute rate, with exactly 1 of 29 derived scopes coming out wider than it should have been, and we report that one rather than rounding it away. The demo shows four live refusals, none of them mocked, including a postmortem contract that reaches nothing at all. On the quality side, one promoted specialist produces one document structure where the same task asked ad-hoc produced four. And we evaluated six alternative designs with measurements rather than asserting our choices were the right ones.

What we learned

Lexical similarity is the wrong instrument for prompt clustering, and we only learned that by watching a Jaccard threshold sit at zero promotions on real data. Cluster linkage is a security decision, not a tuning parameter, because single-linkage chaining merges tasks and a merged cluster produces a scope that is the union of two tasks' permissions. Statefulness in the runtime corrupts the evidence, which is why the shared Codex thread was more dangerous than a crash would have been: a system that learns from observation has to be certain the observation is real. A green test suite is a claim that needs auditing like any other, since a shebang was enough to make our most important tests disappear without a single warning. New code paths need to be tested against the whole pipeline rather than their own logic, or you ship a split path that routes correctly in isolation and unmatched in production. And measurements need their methodology fixed before the numbers are quoted, not after. What's next for Codify

What's next for Codify

The obvious next layer is principal-level scoping, binding contracts to who is asking, not just to what is being asked, so that an operator and a read-only viewer running the same task get different capabilities. Beyond that we want payload-level egress control rather than host-level: allowing a host is still coarse, and the interesting question is what is being sent to it. Multi-tenancy is the requirement that makes both of those matter in a real deployment. We also want to stay honest about the boundary. Codify enforces at the container edge, which constrains what an agent's process can reach, it does not constrain what a model intends, and it is not a defence against prompt injection at the reasoning layer. What we claim is narrower and, we think, more useful: the permissions a task runs under are derived from what that task has actually done, they hold even when the prompt is not understood, and every refusal is on the record.