VerifyAI

The Security Operations Command Center for AI determinism. VerifyAI emits determinism and adversarial telemetry into Splunk. Giving AI devs, Ops, and compliance teams observance over AI behavior.

Comment

Inspiration

AI agents are drifting in production right now. Customers find out before vendors do. There is no SIEM feed for an agent that started returning wrong outputs yesterday. There is no audit trail a compliance team can reference. Observability teams watch infrastructure with a hundred dashboards. They watch AI behavior with nothing.

We thought that was wrong. So we built the SOC for AI agent determinism.

What it does

VerifyAI scores AI agents for determinism and runs adversarial probes against them. Every result becomes a metric in Splunk Observability Cloud. Same schema as any other SLI in Splunk.

Workflows score 90 to 99 percent on discrete-decision tasks. Multi-step arithmetic comes in at 60 to 80 percent. The spread is the thesis on one screen.

Adversarial sweeps run prompt-injection, PII-leakage, and jailbreak probes against the agent. Each finding maps to a compliance control. GLBA 314.4(c). CMMC 2.0 SI.L2-3.14.1. SOX. ISO 27001.

It all flows into Splunk through OpenTelemetry. Engineering, ITOps, and compliance teams query it through MCP. One operational pane.

How we built it

Backend on Modal. Eight FastAPI endpoints, each streaming via Server-Sent Events. DeepTeam v0.2.7 generates adversarial probes through OpenRouter. Ed25519 signs every certificate.

Telemetry pushes directly to ingest.us1.signalfx.com/v2/datapoint with X-SF-TOKEN auth. Two helper functions wired into the sweep endpoints. Silent skip if the token is absent. Never blocks the return path.

The MCP server is the headline. Eight tools exposed to Claude Desktop. Three of those tools internally call the Splunk MCP Gateway as a client. Same process. Two roles. We connect to Splunk's streamable HTTP MCP endpoint, list the 12 o11y tools live, and call o11y_execute_signalflow_program for drift queries. Server and client in one process.

Splunk dashboard auto-deploys via the v2 dashboard API. Five panels. The overall determinism score is one number front and center. Spread by workflow shows the failure modes.

Challenges we ran into

Cisco Deep Time Series and Foundation-sec hosted models live on Splunk Cloud Platform, not Observability Cloud. We learned that mid-build. Rather than spin up a parallel Splunk Enterprise instance under deadline pressure, we documented those as Roadmap and shipped the Observability story end-to-end.

The 8-endpoint Modal free tier cap pushed us into folding three planned endpoints into existing ones. Tight, but it works.

Accomplishments that we're proud of

The composition pattern works end-to-end. VerifyAI writes determinism telemetry into Splunk on one side and reads it back through Splunk MCP Gateway on the other side. Same Splunk instance. Same metrics. Loop closes. We have a live response showing value 0.8446 from an actual Echelor AI Chat workflow flowing the full path.

Four real customer fixtures already in the dashboard. Echelor design partner. NCE Construction. Fifth Third Newline. Twenty-nine historical sweeps replayed into Splunk so the dashboard has shape from minute one. Nothing synthetic.

Adversarial sweep wired into the same metric namespace as determinism. Eight DeepTeam probes against Echelor scored 100 percent blocked on a live run, mapped to four GLBA Safeguards Rule controls. Security signal joined to determinism signal in one pane.

Three architecture diagrams in the repo. Static view, temporal view, composition zoom. Each one shipped as SVG, all rendering inline on GitHub.

The submission is honest. Everything claimed as built is in the code. Everything aspirational is in the Roadmap. No claim inflation.

What we learned

Composition is the killer pattern for MCP. An MCP server that is also an MCP client multiplies leverage. Same protocol, both directions, one process. This is what Splunk MCP Server was built to enable.

Determinism is a first-class SLI. Once it is a metric in Splunk, every detector, alert, and playbook primitive that already exists for error rates and latency just works. The schema is ordinary metric data. SOC teams adopt it without retraining.

What's next for VerifyAI

Cisco Deep Time Series wired into the drift detector. The model lives in Splunk AI Toolkit on Splunk Enterprise. We spin up the trial, point the forecast at verifyai.determinism.score, and catch trend reversals before threshold breach.

Foundation-sec as the adversarial generator. DeepTeam is the placeholder. Foundation-sec is the upgrade. Drop-in once the API is reachable from Observability Cloud tokens.

Splunk AI Assistant natural language queries. "Show me workflows that drifted below 80 percent this week." The SignalFlow programs are already programmatic. The Assistant generates them.

Auto-pause on threshold breach. Splunk detector fires a webhook. Modal control plane pauses the workflow. The webhook receiver is fifty lines.

Auto-page on-call. PagerDuty through Splunk On-Call. Same detector primitive.

Splunk Mobile posture badge. The iframe pattern already ships from Vercel. Splunk Mobile dashboards host the same URL.

Multi-tenant Splunk org. One operator login. Every customer's agent posture underneath. Requires Splunk Cloud Platform tier.

Splunkbase listing. Package the dashboard, the MCP wiring, and the OTel exporter as a Splunk app. One-click install.

Built With

  • deepteam
  • ed25519
  • fastapi
  • model-context-protocol
  • opentelemetry
  • python
  • splunk-mcp-server
  • splunk-observability-cloud
Share this project:

Updates