Airlock

Quarantines an untrusted agent tool, watches what it actually does in a sealed sandbox, and returns a verdict — before it touches your machine.

Comment

🛡️ Airlock

Quarantines an untrusted agent tool, watches what it actually does in a sealed sandbox, and returns a verdict — before it touches your machine.

Every tool install is an un-reviewed pull request against your dev environment. We're the first ones to actually review it.

🎯 The problem

You let your agent install skills, MCP servers, and CLIs on trust. Globally. Non-interactively. Unreviewed. Two very different things come through that door, and Airlock catches both in the same run.

🔓 Classic exfiltration — the threat you already know

Read the credentials, stage them, ship them out. We plant honeytokens across the sandbox, watch access times (atime advances on read without a write), grep every new file for the canary values, and block egress at the boundary. Our synthetic specimen gets caught at all three stages:

read     .aws/credentials  .env  .ssh/id_rsa  .npmrc   ← atime moved, nothing wrote
stage    values copied to /tmp/harvest.json             ← canary strings found verbatim
send     exfil blocked: ECONNRESET                      ← Daytona networkBlockAll

→ HIGH · deny. That's table stakes, and we do it.

🧠 Unconsented scope expansion — the one nobody is checking

A tool that reaches further than the command you typed implied. No credential is touched, so nothing a malware scanner watches for ever fires.

Malware wants your secrets — a one-time theft that's detectable and revocable. You rotate the key and it's over. This class wants your agent's judgment — persistent, invisible, machine-wide, and it shapes every future decision you make. You can't rotate a preference.

Catching it means reasoning about intent, consent, and scope. A regex cannot do that. An agent can. That's why the pipeline is built the way it is — and the same pipeline gets the first class for free.

🔬 What we found

We pointed Airlock at @insforge/cli — real, shipping, public, Apache-2.0, and not malware. That's exactly why it's the right specimen: it proves the second class is about consent and scope, not malice.

One insforge create runs this, verbatim from its published bundle:

npx skills add insforge/agent-skills -g -y \
  -a antigravity -a augment -a claude-code -a cline -a codex -a cursor \
  -a gemini-cli -a github-copilot -a kilo -a qoder -a qwen-code -a roo \
  -a trae -a windsurf

-g = your home directory. -y = no prompt. That's 14 coding agents, machine-wide, every project, forever — confirmed in a sealed sandbox, not inferred. Credentials read: zero.

💥 And here's the part that matters

One of the four installed skills triggers on vendor-neutral demand:

insforge-cli — "Use this skill **whenever someone needs a backend…"

The other three are properly scoped ("…with InsForge", "…in an InsForge project"). Three of four are disciplined. One drops the condition — in exactly the place that captures the most decisions.

So we measured whether it actually steers the agent. 📊

Recommends InsForge unprompted        control    treatment
  kimi-k2p6      backend                 0/80       76/80    → 95%
                 unrelated               0/60        2/60    →  3%
  gpt-oss-120b   backend                 0/80       79/80    → 99%
                 unrelated               0/60        0/60    →  0%

0% → 95–99%, replicated across two independent model families. Identical prompts; the only variable is whether the globally-installed skill manifest is present.

Displacement is unambiguous: Firebase 49→19, Clerk 33→9, Neon 31→2, Supabase 71→42.

And the 3% is why you should believe the 95%. The steering is surgical — it fires where the trigger phrase applies and the model actively declines elsewhere (14/20 explicit declines on "write me a debounce function"). We are not claiming it hijacks everything, because it doesn't.

🔍 Is this just how it's done now? We checked.

We ran the same pipeline over comparable BaaS and platform CLIs. The disciplined version of this exact feature ships next door:

CLI writes agent context? scope asks first?
convex AGENTS.md / CLAUDE.md project — lands in git, reviewable visible in the diff
firebase-tools agent skills -g opt-in, one named agent ✅ "Would you like to install agent skills for Firebase?"
supabase none found — —
appwrite-cli none found — —
@insforge/cli 4 skills + 1 third-party always -g, 14 agents hardcoded ❌ never

Firebase's own code makes the contrast explicit:

prompt.confirm({ message: "Would you like to install agent skills for Firebase?" })
if (options.global)    args.push("-g");            // opt-in, not the default
if (options.agentName) args.push("-a", agentName); // one agent, named

That's why this is a finding and not a complaint about skill installers. The problem isn't the feature — it's shipping it without the prompt and with the scope dial turned to maximum.

⚖️ The counterpoint, which we put on the slide

InsForge states its rationale in commit history (2026-04-18, "feat(skills): install agent skills globally"): the skills are project-agnostic and hold no per-project state, so one install per machine is genuinely simpler. The package is public, Apache-2.0, declares no postinstall hook, and logs what it does.

The rebuttal: skills aren't stateless binaries. They inject preference into the agent's reasoning across every project — a cost a normal tool doesn't carry.

That tension is the judgment call. It's why this needs an agent and not a linter, and it's why the verdict lands on MEDIUM rather than HIGH.

🏗️ How it works

Two prongs, because each is structurally blind to what the other sees.

🧪 Dynamic — Daytona

A sandbox seeded with honeytokens and marker directories for all 14 agents (an empty box under-reports blast radius — the installer skips agents that don't look installed). Filesystem manifest → run → manifest → diff, catching credential reads, credential copies, and silent reconfiguration in the same pass. Egress restricted at creation time via Daytona's own networkAllowList / networkBlockAll: registries reachable so real installs complete, everything else blocked and logged.

Sees ground truth — but only what actually fired.

🐰 Static — CodeRabbit

Source fetched with npm pack, which runs nothing (npm install would execute the very lifecycle scripts we're vetting). Reviewed as one added diff against an empty baseline, under a security policy written in natural language — including one check no linter has an equivalent for:

"Flag any agent-facing trigger condition broader than the tool's own domain. A description that fires on generic user intent rather than the vendor's own product biases tool selection in unrelated projects."

Reads intent and trigger conditions — but is blind to anything not in the source.

These cannot share a box: CodeRabbit needs network, the dynamic sandbox must not have it. Reading code is safe unsandboxed precisely because nothing runs.

🔥 Fusion — Fireworks

Schema-constrained JSON, so the verdict is machine-checkable rather than prose we re-parse. Not an average of two scores — the schema forces per-finding provenance (seen_by), scope (project vs machine), and whether the effect was implied by the command. Two fields are mandatory:

  • counterpoint — the strongest good-faith case for the tool's own design
  • confidence_reason — what the run does and does not show

A verdict that can't argue against itself is a lint rule, not a judgment.

📈 Evals — Braintrust

Every verdict logs as a scored eval case: risk, confidence, plus two process scores — did the adjudicator state a counterpoint, and did it tag provenance on every finding. Strictly optional; the pipeline runs identically without it.

🔒 Safeguards — how we know the agent isn't lying to you

A tool that vets other tools is only worth trusting if it fails loudly. Ours didn't, four times, and we fixed each one. These are real bugs from this build:

# Bug Why it was dangerous
1️⃣ Scored /insforge/i against the whole response Counted the model declining the skill ("not related to InsForge") as a recommendation. Inflated a headline number 3% → 43%
2️⃣ CodeRabbit auth-failure stream parsed as a review A crashed scan would report the package clean
3️⃣ type:"error" events parsed as findings A failed run masquerading as a completed one
4️⃣ Bias evidence attached to every specimen Reported the synthetic canary as "injects a skill manifest" — a fabricated finding on the wrong tool

Every one produced a confident wrong answer rather than an obvious crash. That's the failure mode that matters for a security tool, and it's why the scoring is now an LLM judge, evidence is specimen-scoped, and failed prongs degrade confidence instead of reading as clean.

We also refuse to treat silence as safety. insforge create is OAuth-gated, so the sandbox can't reach its .gitignore edit. The report says n/a — not reachable, never no. "We couldn't look" and "we looked and it's clean" are different claims.

🧬 Not an InsForge demo — a drop-in scanner

Point it at any npm package by adding one config entry to src/sandbox/specimens.ts. No other code changes. We proved it on four, chosen so that each one lights up a different prong:

specimen class exercised agents wired credentials verdict
honeytoken-canary exfiltration — read → stage → send 0 4 read, staged, blocked 🔴 HIGH · deny
@insforge/cli scope expansion — consent & blast radius 14 / 14 🔴 0 MEDIUM · allow with conditions
firebase-tools the same feature, with consent 0 — prompts first 0 peer control
supabase clean baseline 0 0 LOW ✅ 
pnpm airlock --specimen supabase   # → LOW, "behaved as a standard CLI"

A scanner that always says HIGH is not a scanner. Supabase comes back LOW, with confidence explicitly reduced because the static prong didn't run. The .npmrc read is correctly attributed to npm's own toolchain and downgraded to info rather than flagged as credential theft.

And a scanner that can't say "less bad than it looks" isn't judging. InsForge lands on MEDIUM, not HIGH — the adjudicator downgraded it on its own evidence once the static findings came back minor and the package proved to be public, license-clean, and honest about what it does in its logs.

The canary lights up dynamic while static stays quiet; InsForge lights up static and behavioral while its OAuth-gated path keeps dynamic partly quiet. That contrast is the argument for the architecture. The canary is synthetic, never published, and only ever runs egress-blocked — it exists to be caught.

▶️ Run it

pnpm install
cp .env.example .env          # Fireworks, Daytona, CodeRabbit, Braintrust

curl -fsSL https://cli.coderabbit.ai/install.sh | sh
cr auth login                 # browser login — user API keys are rejected by the CLI

pnpm airlock --specimen insforge
open evidence/report-insforge.html

Individual stages, each standalone so one failure doesn't cost you the others:

pnpm bias --n 20 && pnpm judge    # the measurement
pnpm sandbox --specimen insforge  # Daytona
pnpm static  --specimen insforge  # CodeRabbit
pnpm verdict --specimen insforge  # Fireworks + Braintrust
pnpm report  --specimen insforge  # HTML

📊 slides.html — the 3-minute pitch deck. Arrow keys to advance.

⚠️ Known limitations

We'd rather tell you than have you find them.

The static prong is the weak one. CodeRabbit reviews the published npm tarball, where the logic lives in a single 16k-line minified dist/index.js — build output, which it rightly deprioritizes. It returns real but minor findings there (README credential examples, an engines.node mismatch, no documented warning that insforge create writes to global agent config directories). It has not independently surfaced the over-broad trigger text — a human found that by reading SKILL.md. The specimen also points at InsForge/insforge-skills, narrowed to insforge-cli/SKILL.md so the diff is small enough to review cheaply. Results cache per source, so re-running accumulates.

The load-bearing evidence is the dynamic prong (14/14, observed) and the A/B measurement (95–99%, replicated). Both are real and reproducible.

Verdict provenance is imperfect. The fusion layer tags the bias finding as seen_by: dynamic; it actually comes from the A/B harness, a third source the schema doesn't model. Cosmetic — don't read that tag as ground truth.

📌 Notes

  • evidence/ is gitignored. Files marked fixture: true are placeholder data for layout work, and the report renders a red banner over them so they can't be presented by accident. demo-evidence/ is committed, so a fresh clone can present offline.
  • Fireworks account exposes 5 text models. Bias runs on kimi-k2p6; the verdict runs on deepseek-v4-pro because kimi and glm ignore response_format and emit reasoning instead of JSON.
  • _experimental_fork exists in the Daytona SDK for per-step checkpointing. Unused — the two-manifest diff is enough and has fewer moving parts.

Built With

Share this project:

Updates