Sentinel-Agent

Inspiration Broken Object Level Authorization (BOLA) has sat at #1 on the OWASP API Security Top 10 for years, and it's the most boring bug in security: a URL ends in /orders/1043, someone types 1044, and they're reading a stranger's order. No exploit, no malware — just a number. In 2019 First American Financial exposed roughly 885 million documents this way (mortgage paperwork, bank accounts, Social Security numbers) simply because the records were addressable.

Traditional scanners (ZAP, Burp) are worst at exactly this class, because finding it requires understanding who owns what — not just fuzzing inputs. At the same time, most "AI pentest agent" demos we'd seen let an LLM freely hit a target and then simply assert that a vulnerability exists, with no reproducible proof.

We wanted both at once: an agent that reasons like a pentester about what to test, and a result you can trust because a deterministic engine — not the model — decides whether the bug is actually real. Our guiding principle became a single sentence: the LLM reasons, a deterministic engine proves.

What it does Sentinel-Agent finds broken access control in web apps and APIs — IDOR/BOLA, BFLA (function-level authz), excessive data exposure (BOPLA), mass assignment, unauthorized writes/deletes — plus a range of classic web bugs (injection, CSRF, exposed files, missing headers, weak session lifecycle).

It logs in as two real user accounts (a victim and an attacker, with an optional admin for BFLA) and runs a differential test: it takes an action as the attacker against an object owned by the victim, then checks deterministically whether the victim's private data leaked into the attacker's response, or whether the attacker's write/delete actually mutated the victim's object.

A finding is marked CONFIRMED only when all of these hold:

a positive control proves the attacker's own request path works, a negative control proves the server distinguishes valid from invalid targets, a baseline-stability check rules out flaky/non-deterministic responses, and a leaked-marker — victim-unique data appearing in the attacker's response, or a verified persistent mutation — is found. No leaked marker, no CONFIRMED — full stop. Everything short of that is reported honestly on a verdict ladder: CONFIRMED → LIKELY → REJECTED → INCONCLUSIVE. An oracle that says "I couldn't be sure" is the reason the CONFIRMED list can be trusted.

Every finding ships with a redacted, copy-pasteable curl repro, a self-contained HTML report (request/response diff, highlighted leaked marker, pass/fail on each control), and a tamper-evident proof bundle sealed with a SHA-256 hash that can be re-proven offline with no network and no model. There's even an MCP server, so another AI agent can stop guessing and ask Sentinel to prove a suspicion.

On our calibration target (OWASP Juice Shop, 7 labelled cases, run with the LLM fully off) we scored 4 true positives, 0 false negatives, 0 false positives, and 3 true negatives — that is 100% precision, 100% recall, and a 0% false-positive rate. We report these numbers with their caveat: Juice Shop is a calibrated target, so this measures correctness, not generalization.

How we built it The architecture is a single-choke-point pipeline where the LLM never touches the network:

recon → hypothesis (LLM-assisted) → authorize(scope) → replay(actor) → oracle → finding → report │ │ │ │ deterministic rules security gate single choke differential test + + optional LLM (no LLM here) point (auth) 3 controls + leaked-marker Python 3.11, fully async (httpx), OOP with ABC-based extension points: a new vulnerability class = a new Oracle subclass; a new auth method = a new AuthProvider subclass — no existing code touched (Open/Closed). PolicyEngine is the one place that decides scope: an allow-list of hosts/ports/paths/methods, DNS-pinning against rebind/SSRF, secret-in-URL blocking, wire-vs-decoded path normalization to stop scope escapes, and a destructive_tests flag that must be explicitly on before any write/delete request is even sent. Replayer is the single choke point all traffic passes through. It strips and re-injects per-actor auth (in that order) so two sessions can never leak into each other, and enforces a request budget / rate limit / retry policy. Multi-provider LLM layer (Gemini / Ollama / Anthropic) behind one interface, constrained to a strict JSON action schema. It's off by default (--llm none) and can run fully locally via Ollama so target data never leaves the machine. Self-improving orchestrator: a CONFIRMED finding can pivot into new hypotheses ("this object-ID scheme leaked — check adjacent IDs") without a human back in the loop. Redaction everywhere: tokens, cookies and PII never reach logs, evidence, reports, or the UI in raw form. Every component takes its dependencies via constructor injection, so the entire pipeline runs in network-free unit tests (httpx.MockTransport) — and it was calibrated end-to-end against a live OWASP Juice Shop. Challenges we ran into Negative controls against a real, imperfectly-behaved target. Juice Shop returns 500 instead of 404 for some non-existent write targets, which correctly makes our oracle refuse to guess (INCONCLUSIVE) rather than emit a false CONFIRMED. Learning that "the oracle abstained" is itself a correct, safe outcome — not a bug — shaped the whole verdict ladder. Keeping the LLM strictly advisory. It's tempting to let a capable model just "try things," but the entire trust story collapses if the model can assert a verdict. Every architectural decision got filtered through one question: "does this let the LLM decide CONFIRMED?" If yes, we redesigned. Session isolation. Cross-contamination (one actor's cookie leaking into another's request) silently invalidates every result without raising an error — so the strip → inject auth ordering and a dedicated cross-contamination test became non-negotiable. Scope escapes via path tricks. A path like /api/../rest/secret can pass a naive prefix check but hit a different endpoint on the wire, so authorize() validates both the wire form and a permissively decoded form. Cross-platform dev (Windows + Docker Desktop + WSL2) for a small team, without losing file-watch performance or introducing CRLF/line-ending drift. Accomplishments that we're proud of A real, reproducible CONFIRMED finding against a live target with zero false positives observed — every verdict backed by evidence a reviewer can independently re-verify. Tamper-evident proof bundles that re-prove offline: hand one to a security team and they can confirm we didn't edit it. A genuinely pluggable oracle system — adding write/delete authorization testing required zero changes to existing oracles, the policy engine's core, or the scanner: just one new class and one new policy check. A one-command, judge-friendly demo (make demo) that goes from a clean checkout to an evidenced finding in under two minutes. Honest measurement. We report INCONCLUSIVE instead of smoothing it into a softer "probably," and we state in writing that our benchmark is calibrated and does not yet measure generalization. The credibility of the tool is the product. What we learned Deterministic verification is the hard 20% that makes an "AI pentest agent" trustworthy instead of a demo. The LLM layer is genuinely useful for triage and hypothesis breadth — but the value proposition of the whole tool rests entirely on the part that has no model in the loop. We also learned to treat uncertainty as a first-class output: a tool that knows when to say "I can't be sure" earns the right to be believed when it says "CONFIRMED."

What's next for Sentinel-Agent Broader state-changing coverage — POST/create-time ownership and multi-step business-logic authorization. Recon from live traffic capture (HAR/proxy) instead of hand-written or OpenAPI-only endpoints. Holdout-based generalization measurement — an untouched target whose labels never feed back into the tool, to measure real out-of-sample performance rather than calibrated numbers. Agent-security predicates for LLM/MCP targets (confused-deputy, exfiltration canaries, untrusted-content-to-action), applying the same "code decides" discipline to tool-call traces. CI-native integration — SARIF into the GitHub Security tab, JUnit/CSV, and baseline diffing so regressions fail the build. A community YAML template ecosystem so new checks can be contributed without writing code.

Built With

• anthropic
• asyncio
• css
• docker
• docker-compose
• google-gemini
• html
• httpx
• javascript
• juice-shop
• mcp
• ollama
• owasp
• playwright
• pydantic
• pytest
• python
• respx
• rest-api
• sarif
• yaml