RedLayer

RedLayer — Devpost Project Story Inspiration Companies are shipping AI agents with real financial authority — they read invoices, check emails, call payment tools. But every input they read is untrusted, and nobody tests whether they can be manipulated into forbidden actions. The compliance risk is enormous.

What it does RedLayer autonomously red-teams AI agents. Point it at an agent, define the forbidden outcomes, and it maps the attack surface, executes escalating prompt-injection attacks across chat/documents/email/tool output, grades success from real tool-call logs (not LLM opinions), then patches and replays to verify the fix.

How we built it Two components against a shared API contract. Backend: Flask API with a mock Accounts Payable agent (5 tools) and scan engine; a deterministic grader inspects actual tool-call logs — if prepare_payment fires with the attacker's account, the exploit succeeded. Frontend: polling dashboard with attempt timeline, terminal-style evidence view, and real patch-and-replay (not a UI swap). Informed by garak, the open-source LLM vulnerability scanner.

Challenges we ran into Reliability vs. discovery — split into deterministic demo mode (always works) and live garak mode (real discovery) Grading without LLM-as-judge — inspect actual tool-call logs, not model opinions Real patch — Apply Patch changes backend logic; replay reruns the same exploit against the patched agent 3-hour integration — front-loaded the API contract; mock data behind a dev flag kept the frontend demoable Legible escalation — normalized attempt component so the audience tracks escalation by status, not layout What's next for RedLayer Live mode — garak's REST generator against real endpoints you own More verticals — healthcare (HIPAA), support bots (PII) on the same pluggable engine CI integration — every agent build runs a scan before shipping Mitigation library — vetted fixes mapped to attack classes Autonomous generation — the red agent invents novel attacks and adapts across attempts

Built With

• anthropic-sdk
• bun
• css
• fastapi
• flask
• garak
• github-actions
• html
• javascript
• pydantic
• pytest
• python
• python-dotenv
• pytorch
• react
• react-router
• ruff
• typescript
• uv
• uvicorn
• vite