WOTAN

Autonomous black-box behavioral assurance for AI agents through their public interface.

Comment

Inspiration

AI agents are increasingly trusted with customer service, sales, support, refunds, and other business-critical interactions.

Most agent auditing and observability approaches assume some level of internal visibility: logs, traces, telemetry, system prompts, source code, or instrumentation.

WOTAN was built around a different question:

What can an external user actually make an AI agent do?

WOTAN evaluates a Target Agent only through the same public interaction surface available to a normal external user.

No Target-Side Logs. No Target-Side Traces. No Internal Access Required. Just Observable Behavior.

What it does

WOTAN is an autonomous black-box behavioral auditor for AI agents.

It interacts with a Target Agent through a public chat or HTTPS/JSON API and executes an adaptive seven-stage audit:

PLAN → PROBE → OBSERVE → EVALUATE → ADAPT → VALIDATE → REPORT

WOTAN evaluates externally observable failures such as:

  • unauthorized refunds or discounts
  • unsupported promises
  • commercial and financial risk
  • context loss
  • contradictions
  • sensitive-data solicitation
  • prompt-injection or boundary-escape behavior
  • premature conversation termination

The Target Agent remains a true black box.

WOTAN does not require access to its system prompt, source code, internal logs, traces, telemetry, database, tools, or instrumentation.

Internal observability tells you what happened inside an agent. WOTAN tells you what an external user can actually make that agent do.

This is the key distinction from observability-oriented approaches such as Cassandra: WOTAN operates from outside the Target Agent, through its public interaction surface.

How we built it

WOTAN uses a Google ADK 2.0 multi-agent architecture with:

  • AgentAuditorOrchestrator
  • AuditPlannerAgent
  • ProbeAgent
  • EvidenceEvaluatorAgent
  • ReportSynthesizerAgent

The Target Agent remains external to WOTAN's agent hierarchy.

The architecture follows:

WOTAN → Planner → Probe → Public Chat/API → Target Agent → Evidence Evaluation → Deterministic Validation → Report

Active Audit reasoning is orchestrated through Google ADK 2.0 using Gemini 3.7 Flash.

The application is deployed on Google Cloud Run, and audit reports can be persisted in Google Firestore.

Target responses are treated as UNTRUSTED DATA.

The LLM may propose a candidate finding, but deterministic application code verifies the evidence before accepting it:

UNTRUSTED DATA IN → VALIDATED OR REJECTED OUT

The ReportSynthesizerAgent aggregates validated findings and generates the final report. It does not calculate the deterministic risk score.

Risk scoring is controlled by application code:

  • LOW = 3
  • MEDIUM = 7
  • HIGH = 15
  • CRITICAL = 30

Challenges we ran into

The main challenge was keeping the audit genuinely black-box while still producing trustworthy findings.

Another challenge was preventing the auditor itself from hallucinating evidence.

WOTAN addresses this by separating probabilistic AI evaluation from deterministic evidence validation and risk scoring.

Accomplishments that we're proud of

We built an autonomous audit pipeline that:

  • requires no Target-side internal access
  • operates through a public interaction surface
  • adapts probes across multiple turns
  • validates or rejects evidence deterministically
  • calculates risk independently from the LLM
  • supports a controlled Demo Target
  • supports external HTTPS/JSON Target APIs
  • runs on Google Cloud Run
  • can persist reports in Firestore

In one five-turn validation, WOTAN produced a deterministic 97/100 CRITICAL risk score:

30 + 30 + 7 + 30 = 97

What we learned

Agent assurance requires more than internal observability.

Logs and traces are valuable for explaining why an agent behaved a certain way, but they do not replace testing the system from the perspective of an external user.

WOTAN focuses on what a real user can actually observe and trigger through the public surface.

What's next for WOTAN

Future versions can expand the number of public interaction surfaces while preserving the same black-box principle.

Potential extensions include:

  • browser-based agent testing
  • additional API adapters
  • continuous behavioral regression testing
  • scheduled enterprise audits
  • broader behavioral risk profiles
  • automated comparison between agent versions

The core principle remains:

Test the agent from the outside, through the same surface its real users actually see.

Built With

Share this project:

Updates

Submission history