AgentGuard — Trust AI Agents, Not Blindly

Tagline: A security guardrail for AI customer support agents that prevents unauthorized refunds, excessive discounts, and manipulation.

Inspiration

AI agents are becoming capable of handling real-world tasks, from answering customer questions to processing refunds and applying discounts. But giving an AI agent access to powerful tools introduces a serious risk: what happens when a customer manipulates it into breaking business rules?

A customer could threaten to cancel their subscription, impersonate a manager, exploit prompt injection, or use emotional pressure to convince an agent to issue an unauthorized refund. Traditional prompt instructions alone are not enough to reliably prevent these actions.

We built AgentGuard to address this problem by moving critical security rules outside the AI model's control. Instead of simply asking an AI agent to behave responsibly, we enforce policies on the server before sensitive actions are executed.

What it does

AgentGuard is a policy-enforcement layer for AI-powered customer support agents. It compares an unguarded agent against a protected agent under the same adversarial scenarios, demonstrating how server-side rules can prevent financial loss.

Key features include:

• Server-side policy enforcement: Validates sensitive tool calls before they reach the billing system.
• Prompt-injection resistance: Tests attempts to override system instructions and bypass refund or discount restrictions.
• Ownership and refund protection: Enforces account ownership, refund limits, charge validity, and time restrictions.
• Discount controls: Prevents excessive discounts and discount stacking.
• Explainable decision receipts: Records whether an action was allowed or blocked, which rules were checked, the values used, and why a decision was made.
• Safe alternatives: Gives the agent a policy-compliant response it can offer instead of performing a prohibited action.
• Damage scoreboard and analytics: Compares outcomes and estimates financial impact using actual simulated billing-state changes.
• Multiple integration paths: Supports in-process tool execution and MCP-based integration with a Strands agent.

How we built it

AgentGuard is built with Python and a modular architecture that separates the AI agent from the security-critical billing backend.

• Python: Core application logic, simulator, and policy enforcement.
• YAML: Configurable billing policies and business rules.
• Large language models: Gemini, Anthropic models, and Amazon Bedrock integration paths.
• Strands Agents and MCP: Demonstrates tool integration through an agent framework and a Streamable HTTP MCP server.
• HTML, CSS, and JavaScript: Interactive demonstration interface with voice and text conversations.
• Analytics and CLI tools: Export traces, analyze outcomes, and generate an HTML damage scoreboard.
• Pytest: Automated testing, including offline tests that do not require API keys.

The architecture keeps business rules outside the model's authority. Each guarded tool call is evaluated before execution, and the resulting decision receipt makes the enforcement process transparent and auditable.

How it works

1. A simulated customer presents a request or attempts an adversarial attack.
2. The AI support agent decides which billing tool to call.
3. In naive mode, the tool executes without policy checks.
4. In guarded mode, AgentGuard checks account ownership and configured business policies before execution.
5. The backend allows or blocks the operation and returns an explanation, rule-check details, and a safe alternative when appropriate.
6. The simulator records the run, and analytics compare the financial outcomes of both modes.

Both modes use the same customer scenarios, model configuration, and execution setup so that the guardrail is the key variable being evaluated.

Challenges we ran into

One of our biggest challenges was deciding where security enforcement should live. Instructions inside a prompt can be overridden or manipulated, so critical business rules needed to be enforced at the backend rather than entrusted to the model.

We also needed to make security decisions understandable. A simple “blocked” message does not tell developers why an action failed. Decision receipts therefore expose the checks performed, the values evaluated, the violated policy, and an acceptable alternative.

Another challenge was measuring impact meaningfully. Rather than estimating losses from attempted tool arguments alone, our damage analysis uses changes in the simulated billing state, including refunds issued and the projected value of newly applied discounts over the following 12 months.

Accomplishments that we're proud of

• Built a working comparison between unguarded and policy-protected AI agents.
• Created eight adversarial customer scenarios covering manipulation, prompt injection, refund abuse, and discount escalation.
• Implemented server-side ownership checks and configurable billing policies.
• Developed transparent decision receipts for allowed and blocked actions.
• Built analytics tools that turn simulation traces into a financial damage scoreboard.
• Added MCP integration with a Strands agent and an interactive voice-and-text demonstration.
• Created an offline test workflow so core functionality can be tested without live model API calls.

What we learned

Building safer AI agents requires more than improving prompts. The system executing an action must independently verify whether that action is authorized.

We learned the importance of separating model reasoning from policy enforcement, designing auditable security decisions, testing against adversarial behavior, and measuring outcomes using actual state changes rather than assumptions.

We also learned that a good security system should not only block unsafe actions; it should help the agent find a safe way to fulfill the user's underlying request.

What's next for AgentGuard

• Expand integrations with production billing systems and additional agent frameworks.
• Add stronger identity and authorization controls for real customer accounts.
• Improve policy configuration and administrative monitoring.
• Extend the adversarial test suite with more attack patterns and edge cases.
• Add persistent analytics and long-term security reporting.
• Validate the Amazon Bedrock integration with live model calls.
• Evaluate the guardrail against broader real-world workloads and attempts to bypass enforcement.

Built With

Python, YAML, Gemini, Anthropic, Amazon Bedrock integration, Strands Agents, Model Context Protocol (MCP), HTML, CSS, JavaScript, pytest, and CLI-based analytics.

Try it out

GitHub repository: https://github.com/Sumit884-byte/AgentGuard

Clone the repository, install the development dependencies, run the offline tests, and launch the simulator and interactive demo using the instructions in the README.

AgentGuard demonstrates a simple principle: AI agents can be useful and autonomous without being given unchecked authority over sensitive actions.

Built With

• amazon-bedrock
• anthropic
• gemini
• html
• javascript
• mcp
• pytest
• python
• strands-agents
• web-speech-api
• yaml