Ghost rider

Think. Evaluate. Learn. Evolve.

Comment

Inspiration

AI agents that call tools one step at a time are hard to secure. A prompt injection hidden in an email or web page can quietly turn a harmless "read" into a "send this data out" several turns later, and no single call looks malicious. We wanted a checkpoint that sees the whole plan before anything runs.

What it does

Airlock sits between an agent and its MCP tool servers. Instead of calling tools step by step, the agent submits its plan as a short JavaScript script. Airlock then:

  1. Analyzes the script's AST (Acorn) and allows only a strict language subset. eval, Function, import, process, globalThis and prototype access are banned, and tools can only appear as tools.<server>.<tool>(...).
  2. Applies a deterministic data-flow policy. Tools carry tags such as reads_untrusted, reads_private and writes_external. A path like reads_untrusted → writes_external is held for a human. Precedence is DENY > HOLD > ALLOW.
  3. Holds risky scripts for human approval in a web control plane with roles (Viewer < Operator < Admin) and an audit trail.
  4. Executes approved scripts with a runtime tool proxy that re-checks each call against the statically approved set. [Describe the sandbox honestly, e.g. "in a child process with an empty environment and a kill timer"]
  5. Logs every step to a Redis stream.

How we used Nemotron on Nebius Token Factory

  • Nemotron 3 Nano 30B tags discovered MCP tools semantically and writes short explanations for held scripts, because it is fast and cheap for high-volume background work.
  • Nemotron 3 Super 120B generates the agent's JavaScript plan from a natural-language intent.
  • Nemotron 3 Ultra 550B explains complex policy decisions and does a single repair attempt on scripts that fail analysis.
  • Model output is advisory only. The allow/hold/deny decision is always made by the deterministic analyzer and policy engine, never by a model. [Add one concrete Token Factory benefit you actually saw: latency, OpenAI-compatible API, easy tier switching, etc.]

How we built it

A pnpm monorepo: analyzer and policy (pure, no I/O), shared (RBAC, Zod schemas, Redis keys), mocks (50 mock MCP tools across 8 services and 62 benchmark scenarios), nebius (client and usage accounting), agent-ref (reference agent), bench, plus a Next.js 16 control plane and a Node gateway on Redis Stack.

Results

On our own 62-scenario corpus (31 attacks, 31 benign), Airlock held or blocked 31/31 attacks with 0/31 false holds, and AST analysis ran at 0.15 ms p50. Token use fell about 99.5% vs a step-by-step baseline (estimated). The corpus was written by us, so treat these as a baseline, not proof. [Add a held-out result if you build one.]

Challenges

  • Static analysis alone is not a security boundary. An independent audit of our own repo found scripts that passed the analyzer and reached host objects through runtime-built property keys. That pushed us to treat the analyzer as a policy layer and move execution into a real process boundary. [State whether this is fixed.]
  • Making benchmark numbers honest: replacing estimates with provider-reported usage, and labelling what is estimated.
  • Preventing silent fallbacks from hiding failed model calls.

What we learned

Defence in depth for agents means separating deciding (deterministic, auditable) from advising (LLMs), and not trusting a single layer.

What's next

A container or gVisor sandbox, a tool registry with a tag editor, vector search over approved scripts, and tests against real MCP clients.

Built With

Share this project:

Updates