AI Output Gates

Deterministic, fully local CI gates for AI-generated artifacts—with accepted baselines, policy checks, SARIF, and offline evidence packs.

Comment

Inspiration

AI-generated changes can look finished before they are safe to merge. Model self-review is useful, but it is not a stable CI signal. Teams need a deterministic layer that can answer one inspectable question: did this candidate regress from an accepted artifact?

What it does

AI Output Gates records an accepted local baseline, evaluates a candidate with the same deterministic gate, classifies new, resolved, and unchanged findings, applies an explicit checked-in policy, and emits JSON, Markdown, SARIF, and a self-contained offline HTML evidence pack.

The product runs fully locally. It requires no model API key, makes no runtime LLM or external API calls, and sends no artifact data to a hosted service.

How we built it

The CLI is built with Node.js 20 and TypeScript using Node built-ins. Stable SHA-256 fingerprints make baselines and finding comparisons reproducible. SARIF 2.1.0 integrates with code scanning, while the escaped HTML report contains no scripts, remote fonts, analytics, or network assets.

GPT-5.6 and Codex were used during Build Week for design refinement, test-driven implementation, code review, adversarial edge-case analysis, documentation, and verification. The shipped product remains deterministic and fully local.

Challenges

The hardest work was making evidence trustworthy: normalizing paths without merging distinct findings, preserving deterministic ordering, validating baseline fingerprints, writing evidence atomically, protecting existing directories and symlinks, and keeping the original run, loop, and explain commands backward compatible.

Accomplishments

  • 70 automated tests pass in the repository and in a clean clone of the public v0.2.0 tag.
  • A deliberate regression fails at 22/100 with a delta of -78.
  • The accepted README passes at 100/100 with a delta of 0.
  • The package works directly from the public GitHub tag.
  • Desktop, mobile, keyboard, contrast, secret, and offline-asset checks pass.

What we learned

Models are excellent collaborators for building developer tools, but merge gates need deterministic, inspectable policy. Keeping the model out of the runtime makes the decision reproducible, private, and easy to audit.

What is next

Next steps include artifact-specific policy packs, optional signed baselines, richer CI annotations, and integrations that preserve the same local-first trust boundary.

Built With

Share this project:

Updates