Bastion

Inspiration

Security teams drown in CVE alerts. Every dependency scanner tells you what's vulnerable — almost none tell you what's actually safe to fix automatically versus what needs a human's eyes first. That gap is where real incidents happen: either nothing gets patched because the volume is overwhelming, or something gets auto-merged that shouldn't have been.

I wanted to build something that closes that gap without making the mistake of letting an LLM be the one deciding what's safe. The model can propose. It shouldn't get to decide.

What it does

Bastion is a GitHub App that:

1. Scans a repo's dependency manifest on every push
2. Resolves dependencies to known CVEs via OSV.dev
3. Enriches each finding with an EPSS exploit-probability score and CISA KEV (Known Exploited Vulnerabilities) status
4. Has GPT-5.6 propose a real, minimal patch — a single-dependency version bump with a unified diff and a plain-language rationale
5. Runs that proposal through an OPA/Rego policy — not the model — which decides the outcome:
   • Auto — patch-level fix, actively exploited, high EPSS: opens and merges the PR automatically
   • Ask — real risk but not high-confidence enough: opens a PR, emails the maintainer with the full risk picture
   • Block — major version bump, or low signal: opens an issue with the reasoning, no code touched

The autonomy boundary lives entirely in policy code you can read, audit, and change — not in a prompt.

How we built it

I used Codex throughout — both Codex Cloud and Codex CLI.

Codex Cloud built the first module, the manifest scanner, end-to-end from a single prompt: parsing package.json, requirements.txt, and pyproject.toml, plus its own test fixtures, in one PR.

Partway through the build, Codex Cloud started hitting platform-wide instability — tasks failing with opaque errors during Build Week's peak load. Rather than keep fighting a flaky remote sandbox, I switched to Codex CLI running locally for the rest of the build: the OSV/EPSS/KEV enrichment clients, the GPT-5.6 fix-proposal module, GitHub App actions (branch, PR, merge, issue), the webhook receiver, and the pipeline orchestration tying it all together. Same agent, same reasoning model, just a more reliable loop once local execution was available.

policy/gate.rego — the actual decision logic — I wrote by hand. That's the one piece I wasn't willing to hand off; it's the whole point of the project.

Every module shipped with real test coverage, verified against actual pytest output before merging — never taken on faith from a diff that "looked right."

Challenges we ran into

• Codex Cloud's sandbox has a restricted network — it couldn't reach openpolicyagent.org to install the OPA binary needed to test the policy engine. Rather than fight the sandbox, I made the gate tests skip gracefully when OPA isn't present locally, and added a GitHub Actions CI workflow that installs OPA properly and runs full verification on every push — better coverage than forcing it into a restricted environment anyway.
• A scanner/pipeline desync: an early Codex Cloud PR never actually got merged, so later local work was silently building against a master missing its first module. The pipeline appeared to run but silently processed zero dependencies — caught by adding real logging at the pipeline's entry point instead of trusting a clean HTTP 200.
• A broken control-flow path: a refactor introduced a regression where the "no confident fix" case crashed instead of degrading to a block decision. Caught by insisting on real pytest output rather than accepting "the code looks correct on read" — which is exactly how the bug got introduced in the first place.
• Duplicate issues and branch collisions: repeated pushes were opening duplicate GitHub issues for CVEs already tracked, and two different CVEs resolving to the same target version collided on branch creation. Both fixed with real regression tests, not just patched and hoped.
• API billing vs. product credits: OpenAI's Codex credits and raw API billing turned out to be two separate pools — the $100 Codex credit grant doesn't cover direct API calls. Rather than spend outside the free hackathon allowance, I swapped the in-app fix-proposal call to Groq's free tier (openai/gpt-oss-120b, OpenAI-compatible API) — a clean client-config swap, no logic changes, fully covered by tests including its structured-output fallback path.

Accomplishments that we're proud of

• A genuinely end-to-end working pipeline — not a demo stub. Real GitHub App, real webhook, real OSV/EPSS/KEV API calls, real GPT-5.6-authored patch diffs, real PRs and issues opened against a real repo.
• The policy boundary actually holds under real data: a batch of live CVEs across requests, urllib3, pyyaml, and Pillow produced a genuine mix of auto, ask, and block outcomes — not a cherry-picked happy path.
• 53 passing tests across every module, all real assertions against real (mocked-network, real-logic) behavior — no test written just to make a number go up.
• Catching and fixing every bug above using actual command output as evidence, every time, rather than accepting a plausible-sounding explanation.

What we learned

Mostly: don't trust a "should work" from an agent — or from myself — without actual output proving it. Every real bug in this build was caught by insisting on pasted pytest results or live logs instead of accepting a plausible-sounding explanation. That discipline is honestly the most transferable thing I'm taking away from this week.

What's next for Bastion

• Consolidating multiple CVEs on the same package into a single PR instead of one-per-CVE
• Broader ecosystem support (Go, Rust, Java manifests)
• A dashboard view of pending "ask" decisions across all installed repos

Built With

• cisa-kev
• codex
• docker
• epss
• fastapi
• github-apps
• gpt-5.6
• groq
• opa
• osv.dev
• pygithub
• python
• rego
• render
• resend