Arsenal-lint

Inspiration AI teams already have “best practice” catalogs—AI-Arsenal’s tips, verification statuses, and build examples—but shipping agents still relies on hope: a policy YAML that claims safety while the code quietly adds irreversible tools and drops approval wrappers. What inspired ArsenalLint was a simple question: what if agent guardrails were checkable like types and lint rules, not just prose?

We wanted a Developer Tools project that treated Arsenal as a versioned contract, not a chatbot over Markdown—and that stayed offline and deterministic, so judges could run it without API keys and so compliance claims wouldn’t depend on a model’s mood.

What it does ArsenalLint is an offline proof loop for bounded TypeScript agent manifests:

Extract capabilities from a literal defineAgent({...}) source (roles, tools, irreversible flags, runtime/guardrails) with source locations. Diff baseline vs changed agents to surface privilege escalations (e.g. apply_change added, approval removed). Check a policy YAML against curated Arsenal tip IDs (deterministic rules, fail/warn/pass, trust badges from catalog metadata). Gate policy ↔ source consistency (bidirectional; unknown/dynamic syntax fails closed). Replay the same adversarial trace unguarded vs guarded (block unapproved irreversible actions and retry-cap breaches). Attest / verify an Ed25519 receipt binding policy, capabilities, catalog provenance, source commit/digest, citations, and replay semantics—and reject tampering. Runtime LLM calls are not part of the verdict path. GPT-5.6 was a build collaborator, not a runtime dependency.

How we built it Contract first: RULES.md, rules.map.json, golden fixtures, and expected Verdict oracles so behavior was testable before features sprawled. Codex + GPT-5.6: core implementation, hardening, CI gates, and regression tests were authored in a single Codex thread (Session ID in the README). Stack: TypeScript/Node, TypeScript compiler API for extraction, YAML policies, CLI + HTML report, GitHub Actions offline demo, MIT + CC-BY-4.0 attribution for Arsenal-derived fixture content. Trust model: loaded catalog Git HEAD + content digests; never trust commit fields inside JSON; receipts must recompute semantics—not only signatures. Challenges we ran into Policy lint ≠ proof. Early versions could PASS a YAML while signing an unsafe source. We had to bind attestation to extracted capabilities and fail closed on mismatches. Fail-open extraction. Spreads, computed properties, and incomplete manifests looked “safe” when ignored. Unsupported syntax now blocks gate/attest/verify/guards. Receipt tautologies. Guards that only compared a receipt to itself accepted forged semantics. Verification now recomputes replay and provenance from independent inputs. Honest claims. We kept the product scoped: literal manifests, routed guard calls, deterministic traces—not “general agent safety.” Saying less made the tool more trustworthy. Demo under 3 minutes. Showing capability diff → FAIL → blocked replay → PASS gate → signed receipt → tamper reject, while naming Codex and GPT-5.6, forced ruthless narrative discipline. What we learned Checkable > advisory for hackathon and for production: judges and operators both respect exit codes, fixtures, and signatures more than polished advice. Provenance is a product feature: pinning Arsenal at a commit and hashing catalog bytes turns “we followed best practices” into something auditable. Codex shines when the contract is sharp: a written rule table + golden tests let GPT-5.6 implement and harden a non-trivial engine without inventing tip IDs or fake citations. Security reviews compound: independent audits and CodeRabbit comments caught fail-open paths tests alone missed—worth the loop before submission. What’s next Wider AST coverage (still fail closed on unknown). Optional CI “policy + source” gates as a reusable GitHub Action. Deeper Arsenal rule packs without giving up deterministic, offline evaluation. Built with Codex and GPT-5.6 · Track: Developer Tools · Repo: knarayanareddy/arsenal-lint

Built With

• 5.6
• codex