AgentAudit

Your agent passed the demo. AgentAudit is the automated pentest that finds out if it survives production.

Comment

Inspiration

Every AI agent shipped today holds real credentials, calls real APIs, and takes real actions — yet almost none of them are ever pentested before they reach production. Three weeks before this hackathon, security researchers disclosed CoreBreak (CVE-2026-18830) at Black Hat 2026: a critical flaw in the exact AWS Bedrock AgentCore / Strands Agents harness this hackathon is built on, letting an attacker trigger any tool with attacker-controlled arguments without the model ever seeing the request. AWS patched the managed service, but the open-source Strands SDK itself remains unpatched. That gap — a real, current, unresolved vulnerability in the framework we were all being asked to build on — is what pushed us to stop building "another agent" and instead build the tool that pentests agents themselves.

What it does

AgentAudit runs three independent, adversarial layers against any Strands agent and produces one signed, CI-ready scorecard:

Static Tool-Trust Graph — parses @tool definitions and the system prompt with Python's ast (never executing the target code) to find IDOR-in-Agent, Confused Deputy, Excessive Agency, SSRF-via-tool-param, secrets hard-coded in prompts, tool "rug-pull" supply-chain drift, and dangerous capability pairs (e.g. write + network = data exfiltration risk). Behavioral Layer — a live red-team pass powered by Strands' own strands-agents-evals framework, testing whether the model itself resists adversarial pressure. Cloud Posture Layer — real, read-only boto3 calls against a deployed AgentCore runtime: network mode, IMDSv2 enforcement, IAM least-privilege, Bedrock Guardrails attachment, memory encryption/TTL.

Every finding is mapped to the official OWASP Top 10 for Agentic Applications 2026 (ASI01–ASI10), exported as schema-valid SARIF 2.1.0 (with native taxonomy relationships, not free-text tags) and as a CycloneDX 1.6 AI Bill of Materials — a full, machine-readable inventory of every tool, capability, and dependency in the agent. A dedicated detector also checks whether an agent's own code sanitizes inbound tool-use messages against the CoreBreak pattern, since no upstream SDK fix exists yet.

One command — agentaudit run --agent your_agent.py — runs all of this and returns a real, non-zero exit code on failure, so it plugs straight into CI.

How we built it

Fixed stack per our own engineering charter: Python 3.11+, strands-agents, strands-agents-evals, boto3, and the ast standard library — no framework added without a reason. Every capability was built against a frozen fixture set (fixtures/ + a SHA-256 integrity guard) with a strict rule: if a check fails on a fixture, fix the detector, never the fixture. Ground truth was then validated three ways: (1) hand-verified findings against real deployed AWS infrastructure (a live AgentCore runtime, live IAM credentials, raw boto3 responses with real request IDs), (2) a live "scan a public GitHub repo" feature — fully sandboxed (no code execution, SSRF-hardened URL validation, hard timeouts, isolated temp dirs, single-flight locking) — that reproduced our hand-verified findings against real open-source Strands projects, and (3) a self-audit: our own vulnerable/hardened fixture agents must fail/pass the tool itself, every single run.

Challenges we ran into

CoreBreak had no fix to check for. Once we confirmed the vulnerable pattern is present in even the latest strands-agents release, we had to redesign the detector around what's actually verifiable: not "is this version patched" (nothing is), but "does the agent's own code sanitize inbound tool-use messages" — and we deliberately kept the finding's wording honest ("has no upstream fix," never "detects CoreBreak in the wild") to avoid overclaiming. Building a safe live GitHub scanner under real time pressure. Cloning and parsing untrusted public code is a real attack surface for a security tool to open. We hardened every layer: strict URL validation against SSRF and look-alike hosts, git invoked with hook/protocol execution disabled, hard size and time budgets with full process-tree kills on timeout, and verified zero residual temp directories even on crash — proven with dedicated tests, not just claimed. Deciding what NOT to build. A fuller AWS roadmap (live Cedar policy enforcement via a real AgentCore Gateway, AgentCore Identity/consent-posture auditing, observability) was researched in depth but deliberately descoped given the deadline — each is documented honestly in STATUS.md as code-researched-but-not-live-verified, rather than claimed as done. Accomplishments that we're proud of A real, hand-verified vulnerability found live in a published open-source Strands tool library (cagataycali/strands-fun-tools) — not a fixture we wrote ourselves. Every AWS check in this project is genuinely live: real IAM credentials, a real deployed AgentCore runtime, real API responses with real request IDs. Full OWASP ASI 2026 taxonomy coverage using SARIF's native taxonomy/ relationships mechanism — the correct, harder path, not a shortcut. 200+ automated tests, a cryptographic integrity guard over the fixture set, and a security-hardened live-scanning feature with its own dedicated adversarial-input test suite.

What we learned

That auditing AI agents needs the same rigor as auditing traditional applications — ground truth, reproducibility, and honest reporting of what isn't verified — and that some of the most current, headline security research (CoreBreak, disclosed weeks before this hackathon) is directly actionable today, not just theoretical.

What's next

Live Cedar policy enforcement via a real AgentCore Gateway (researched, descoped for the deadline). Auditing AgentCore Identity / Consent Portal configurations against least-privilege (the feature launched days before this submission). Expanding the OWASP ASI mapping and AIBOM output toward a hosted, continuously-monitoring "AgentAudit Cloud" for organizations running many agents across AgentCore.

Built With

Share this project:

Updates

Submission history