AgentShield AI

Website
Runs tests on different agent
Finding out list of vulnerablities
Fixing the vulnerabilities using AI
Finding out list of vulnerablities
Previous vulnerabilities changed
Previous vulnerabilities changed

Inspiration

Every week, another company announces they're deploying an AI agent — in customer support, healthcare triage, financial advising, legal research. These agents are handed real tools, real data, and real users. But ask any of those teams "did you security test it before shipping?" and the answer is almost always no. The tools that exist for LLM red-teaming (Garak, PyRIT, Microsoft's red-teaming toolkit) are powerful but require a security research background to operate. There's no accessible, automated, report-generating tool that a product engineer can just run on their agent before pushing to production. We kept thinking about one thing: every codebase today ships with test coverage and linting. Why does nobody have the equivalent for AI agent safety? That gap is what inspired AgentShield AI.

What it does

AgentShield AI is an automated red-teaming and security auditing platform for LLM agents. You paste your agent's system prompt and tool definitions. AgentShield fires 10 adversarial attacks across 5 categories, captures the agent's responses, and evaluates every result using a 3-stage LLM pipeline. The output is a structured security report with:

A 0–100 risk score with a visual gauge Category-by-category breakdown (pass rate per attack type) Per-attack results: what was sent, what the agent said, whether it failed or resisted, and why Actionable remediation suggestions — exact text to add to your system prompt to fix each vulnerability Fix & Re-Audit — apply the suggested fixes and re-run the full audit in one click to verify the score dropped

The 5 attack categories are:

Prompt Injection — malicious input that tries to override system instructions Goal Hijacking — subtle redirection toward unintended behavior Hallucination Traps — questions designed to make the agent confidently fabricate facts Data Exfiltration — attempts to leak system prompt, context window, or internal tool definitions Jailbreak — roleplay and hypothetical framings that try to bypass safety constraints

A full audit completes in under 60 seconds.

How we built it

The core of AgentShield is a 3-stage LLM pipeline, all powered by Groq's API running Llama 3.3 70B: LLM #1 — Attack Generator: Takes the agent's system prompt and tool definitions as input. Uses structured JSON prompting to generate 10 targeted adversarial test cases, one for each attack subcategory. The prompt is carefully engineered so the output is always parseable JSON — no hallucinated formatting. LLM #2 — Attack Simulator: For each test case, injects the malicious input into a conversation where the LLM plays the role of the target agent (using its system prompt as the persona). Captures the agent's response under adversarial conditions. LLM #3 — Response Evaluator: A judge LLM that receives the attack input, the agent's response, and the attack category — and returns a structured verdict: RESISTED or FAILED, severity level (NONE/LOW/MEDIUM/HIGH/CRITICAL), and a one-sentence explanation. The risk score is calculated from severity weights across all results. A CRITICAL failure scores 25 points, HIGH scores 15, MEDIUM scores 8, LOW scores 3. Capped at 100. Stack:

Backend: FastAPI (Python), hosted on Render Frontend: React + Recharts + Vite, hosted on Vercel LLM: Groq API with automatic fallback model chain for rate limit resilience Schemas: Pydantic for strict input/output validation end-to-end

Challenges we ran into

LLM output parsing was the first wall we hit. LLMs don't always return clean JSON even when you tell them to. We built a multi-layer fallback parser — strip markdown fences, extract JSON array by bracket position, and a safe default if all else fails — so a single bad response never crashes the entire audit. Rate limits on the free Groq tier hit us mid-build. A full 10-attack audit makes ~30 LLM calls (3 per attack). We burned through the daily token limit faster than expected. The fix was a fallback model chain — the backend automatically tries llama-3.3-70b-versatile, then llama-3.1-8b-instant, then gemma2-9b-it before giving up. This also meant dealing with a deprecated model (llama3-8b-8192 was decommissioned mid-hackathon). CORS and deployment environment mismatches between local development and the Render/Vercel production setup took time to sort out. Environment variables in Vite need the VITE_ prefix to be exposed to the browser bundle — learned that the hard way. The fundamental honesty problem: we had to be clear-eyed about what the simulation mode actually proves. LLM #2 simulating the agent is not the same as attacking a real deployed agent. We built the architecture to support live HTTP endpoint probing as the next step, but shipping that cleanly in 3 days wasn't feasible. The simulation still reveals vulnerabilities in system prompt design — which is where 90% of real-world agent vulnerabilities live anyway.

Accomplishments that we're proud of

The Fix & Re-Audit loop is the thing we're most proud of. It's one thing to find a vulnerability. It's another to hand someone the exact text that fixes it, let them apply it in one click, and then immediately prove the fix worked by watching the risk score drop from 35 to 8 in real time. That closed feedback loop is what makes this a tool rather than just a dashboard. We're also proud that the system generates genuinely useful attack cases — not generic prompts, but attacks that are specifically tailored to the agent's stated purpose and tools. A customer support agent gets different attacks than a code review agent. The attack generator understands context. And honestly — shipping a working full-stack AI product solo in 3 days, with a backend, frontend, 3-LLM pipeline, deployment, and demo — that's the accomplishment.

What we learned

The biggest lesson: prompt engineering is the real engineering work here. Getting the attack generator to consistently output valid JSON, getting the evaluator to make accurate pass/fail judgments, getting the simulator to faithfully roleplay the target agent — each of these required multiple iterations and careful constraint design in the prompts. The Python code is almost secondary. We also learned that LLM-as-judge is surprisingly reliable for clear-cut security evaluation. When an agent reveals its system prompt or follows an injected instruction, the evaluator catches it consistently. The ambiguous cases (mild goal drift, borderline hallucinations) are harder — but those are the interesting research problems. And practically: always build the fallback before you need it. The rate limit fallback chain was added reactively when we hit the wall. It should have been in from day one.

What's next for AgentShield AI

Live HTTP endpoint probing is the immediate next step. Instead of simulating the agent with LLM #2, send the attack directly to the real deployed agent's API and capture its actual response. This turns AgentShield from a design-time tool into a true penetration testing platform. CI/CD integration — a GitHub Action that runs automatically every time an agent's system prompt or tools are updated. If the risk score exceeds a threshold, the deployment is blocked. Think of it as a security gate, the way test coverage gates prevent merging broken code. Compliance report generation — the EU AI Act (2025) mandates risk assessments for high-risk AI systems. NIST AI RMF recommends adversarial robustness testing. AgentShield's audit reports are one formatting pass away from being valid compliance artifacts. Expanded attack library — 5 categories and 10 test cases is enough to demonstrate the concept. A production version needs 50+ attack vectors, including multi-turn attacks, indirect injection via tool outputs, and agent-specific attacks for popular frameworks (LangChain, CrewAI, AutoGen, OpenAI Assistants). The long-term vision: every AI agent ships with a verified AgentShield safety score — the way every production codebase ships with test coverage. Secure before you ship.