Inspiration
As AI agents become increasingly capable of interacting with external tools, APIs, files, and services, ensuring that their actions are safe and trustworthy has become an important challenge. An AI agent may unintentionally execute destructive commands, expose sensitive information, or fall victim to prompt injection attacks hidden in untrusted documents.
This inspired us to build an AI Agent Action Guardrail, a security layer that monitors and evaluates every action before it is executed. Our goal was to make autonomous AI agents safer, more transparent, and more accountable.
What it does
The system acts as an intermediate security layer between an AI agent and its tools. Every proposed tool call is intercepted and evaluated using four independent security signals:
- Policy Check: Identifies potentially dangerous operations, destructive commands, restricted file paths, and payment-limit violations.
- Context Analysis: Detects sensitive information such as personally identifiable information (PII) and secrets, while evaluating whether data is being sent to internal or external destinations.
- Behavior Analysis: Identifies unusual actions by comparing tool usage against the agent's expected behavioral baseline.
- Nemotron Security Judge: Uses NVIDIA Nemotron to evaluate the agent's intent, detect prompt injection attempts, identify possible data exfiltration, and generate explanations for security decisions.
The signals are combined through a weighted risk-fusion mechanism to produce one of three decisions: ALLOW, REVIEW, or BLOCK. Each decision is recorded in a SHA-256 hash-chained audit log to provide tamper-evident records and traceability.
The project also includes an interactive Streamlit demonstration in which an AI agent plans tool calls using a task and an untrusted document. It demonstrates how a malicious prompt injection can influence an agent's proposed actions and how the guardrail intercepts and blocks unsafe operations. All demonstration tools are sandboxed mocks.
How we built it
We developed the application using Python and Streamlit, with NVIDIA Nemotron integrated through the Nebius Token Factory OpenAI-compatible API.
Nemotron serves two purposes: it acts as the agent planner and as the security judge. We implemented independent policy, context, and behavioral checks, followed by a weighted risk-fusion mechanism that combines their results. We also implemented a SHA-256 hash-chained audit mechanism to maintain the integrity and traceability of security decisions.
The application supports a rules-only mode when an API key is unavailable and enables the Nemotron-based security evaluation when API access is configured. We also included automated offline tests to validate core guardrail functionality.
Challenges we faced
One of the major challenges was detecting malicious or unintended actions without unnecessarily blocking legitimate agent operations. We addressed this by combining deterministic policy checks with contextual and model-based evaluation.
Another challenge was identifying prompt injection attacks hidden inside untrusted documents, where malicious instructions could influence an agent's planning process. Integrating Nemotron as a security judge helped evaluate whether proposed actions aligned with the original user task.
We also focused on making decisions explainable rather than simply producing a risk score. The weighted risk-fusion mechanism exposes individual signal contributions, while the hash-chained audit log helps preserve decision history.
What we learned
Through this project, we gained practical experience in AI agent security, prompt injection detection, LLM integration, risk assessment, and secure tool execution. We learned how to combine rule-based systems with large language models to build a more reliable security architecture.
Most importantly, we understood that securing an AI agent requires more than checking its final response. Every action and tool call must be evaluated before execution.
Our project demonstrates an approach toward building safer, explainable, and auditable autonomous AI systems.
Built With
- ai-agent-security
- audit
- behavioral-analysis
- contextual-analysis
- hash-chained
- large-language-models-(llms)
- nebius-token-factory
- nvidia-nemotron-3-super-120b-a12b
- openai-compatible-api
- prompt-engineering
- python
- risk-based-decision-engine
- sha-256-hashing
- streamlit
