AgentProbe

• 

  home

• 

  Regression scenarios

• 

  Register agent

• 

  failed vs passed scenarios

• 

  detailed report

Inspiration

AI agents are powerful, but testing them is still manual and unreliable.
AgentProbe was built to find hidden failures before they reach production.

What it does

AgentProbe attacks AI agents with adversarial scenarios and captures execution traces.
It detects failures, groups them, generates regressions, and compares patched versions.

How we built it

We used FastAPI, PostgreSQL, Redis, RQ, Ollama, and Next.js with React JSX.
The system combines deterministic checks with semantic LLM evaluation.

Challenges we ran into

Reliable evaluation was harder than generating attacks.
We also dealt with tracing, version comparison, and third-party agent integration.

Accomplishments that we're proud of

AgentProbe completes the full attack → failure → regression → rerun workflow.
It also supports testing external Hugging Face Gradio agents.

What we learned

Deterministic checks are more reliable than using LLM judges for everything.
Good agent testing depends heavily on observability and reproducibility.

What's next for AgentProbe

We want to add more agent adapters, CI integration, and historical reliability tracking.
The goal is to make adversarial testing a normal part of shipping AI agents.

Built With

• agentprobe
• ai
• aiagents
• aisecurity
• docker
• huggingface
• javascript
• llm
• ollama
• postgresql
• python
• react
• redis