IFF Sentinel — Automated Auth Security Testing

IFF Sentinel actively probes live OAuth, API key, and session auth flows for real vulnerabilities, then uses a fully offline local AI model to explain and fix them before they ship to production.

Comment

A note before I start

I'm not a software engineer by training — I'm an avionics engineering student, and this is my first hackathon project ever. Everything here was built by leaning hard on a disciplined workflow (AI as architect/auditor, an AI coding agent as implementer, me driving the review loop) rather than years of prior software experience. I wanted to say that upfront because a lot of the "what I learned" section below is as much about learning to build software rigorously as it is about auth security specifically.

Inspiration

The idea came from a real bug, not a hypothetical one. While debugging an AI coding agent's implementation of an OAuth 2.0 flow for another project, I found the agent had shipped code that looked correct — it passed review at a glance — but was missing the access_type=offline&prompt=consent parameters and had a broken authorization code exchange. It was a quiet, dangerous kind of bug: the kind that doesn't fail loudly in testing, it fails silently in production against a real attacker.

That got me thinking about a bigger pattern. AI coding agents are now writing huge amounts of authentication code — OAuth flows, API key handling, session management — fast and with total confidence. But auth code is exactly the category where "looks right" and "is right" diverge the most. Most developers using these agents don't have a security background to catch the gap themselves, and manual review doesn't scale with how fast agents produce code. IFF Sentinel exists to be the adversarial second opinion that catches this before it ships. Coming into this without a traditional CS background actually sharpened that motivation — if I couldn't easily tell whether AI-generated auth code was safe, I assumed a lot of other builders in the same position couldn't either.

What it does

IFF Sentinel — the name is a nod to Identify Friend or Foe, the military/aviation concept of actively challenging an unknown contact rather than passively observing it (a term I actually first came across in avionics, not security) — is an adversarial authentication tester with three parts:

  1. A deterministic Rust probe engine that actively attacks a running auth implementation with 14 real vectors across OAuth 2.0/OIDC, API key auth, and session auth: state replay, PKCE stripping, redirect URI manipulation, token leakage via Referer headers, refresh token reuse, tampered ID tokens, scope escalation, missing rate limiting, error leakage, session fixation, missing CSRF protection, weak token entropy, and incomplete logout invalidation.
  2. A fully offline local AI analysis layer — Qwen2.5-Coder-7B-Instruct, running entirely on-device via llama.cpp with GPU offload — that turns raw pass/fail probe data into a human-readable report: what's exploitable, why, and exactly how to fix it. Nothing about the target's code or vulnerabilities ever leaves the machine.
  3. An interactive visualizer that renders the auth flow as a diagram, color-coded red or green per stage, with a before/after toggle so you can watch a real vulnerability go from exploitable to fixed.

The demo runs against J.A.R.V.I.S., an AI assistant project I built and fully own, tested with complete authorization and zero third-party code or credentials involved.

How I built it

I worked in a strict plan → review → implement → audit loop the whole way through: Claude acted as architect and adversarial auditor, and an AI coding agent (Antigravity) handled implementation, with nothing moving to the next stage until the prior one passed review. Given that I was learning as I went, that discipline mattered more than it might for an experienced team — it caught real bugs before they became load-bearing, including a case where the "AI-generated" analysis was quietly falling back to a fixed template dictionary because a 3-second HTTP timeout was too short for real model inference, and another where a probe's core network call was fabricating its own test data instead of actually testing the target.

The architecture is deliberately three separate, independently-auditable layers: the Rust probe engine has zero AI dependency and zero fabrication tolerance, the analysis layer has zero network access outside 127.0.0.1, and the visualizer is a single dependency-free HTML file. Each layer only trusts a strict JSON contract from the one before it.

Challenges I ran into

  • Proving the AI was actually the AI. Early on, the "AI-generated" reports were secretly coming from a placeholder template. I built a generated_by provenance field into every finding — self-reported straight from the model server's own response — so the report itself proves whether it came from real inference or a fallback, rather than anyone having to take that claim on faith.
  • Windows + Rust + no admin rights. Compiling the probe engine on my dev machine hit a wall: no C++ linker (link.exe not found), and the standard fix (Visual Studio Build Tools) needs admin elevation I didn't have in that environment. I ended up installing a fully portable MinGW toolchain (w64devkit) into user space and cross-compiling for the GNU target instead.
  • Running a 7B model on a 6GB laptop GPU. With a mixed-speed RAM setup (repurposed from an older laptop, since I couldn't afford new RAM), I had to tune GPU offload layers carefully (-ngl 33 turned out to fully offload all 29 layers while leaving headroom for the context window) to get real-time inference working at all.
  • Getting a real before/after story without real production access. Rather than fake it, I gave the tool a real repo I own, with a real vulnerable commit and a real fixed commit — so the demo's "before" and "after" states are genuine git history, not synthetic data.

What I learned

That adversarial review — even self-imposed — catches things "it works when I tested it" doesn't, and that this discipline is learnable even without a traditional software background. Several of the bugs above would have shipped invisibly if I hadn't built in an explicit audit step at every stage and demanded proof over description each time. I came out of this knowing a lot more about auth security, but honestly the bigger takeaway was learning how to actually manage a build like this rather than just write code for it.

What's next

  • CI/CD integration as a GitHub Actions bot that comments directly on pull requests
  • Support for SAML 2.0 and WebAuthn/passkey flows
  • A native Rust (candle) inference path as an alternative to the llama.cpp server, for a fully single-binary deployment
  • A no-code GUI that lets you point IFF Sentinel at a running local or staging instance — entering a URL and auth config through a form instead of hand-editing target.toml and the CLI — making the same live adversarial probing accessible without needing to touch Rust, Python, or a terminal

Built With

  • ai-agents
  • authentication
  • cli
  • cuda
  • cybersecurity
  • devsecops
  • gguf
  • gpu-inference
  • html
  • javascript
  • llama-cpp
  • llm
  • local-ai
  • oauth2
  • offline-ai
  • penetration-testing
  • python
  • qwen
  • reqwest
  • rust
  • security
  • serde
  • svg
  • wiremock
Share this project:

Updates

Submission history