AEGIS-IR — Splunk Hackathon Submission

Autonomous AI agent that investigates security incidents through Splunk MCP — from alert to response in 60 seconds, with zero hallucinations.

Inspiration

A Splunk alert fires at 2 AM. The SOC analyst spends the next four hours manually running queries, copying timestamps between tools, pivoting across data sources, and writing up findings. By the time the investigation is done, the attacker has been inside the network for half a day.

The problem isn't detection — Splunk is excellent at that. The problem is what happens AFTER detection. Investigation is slow, manual, and error-prone. Junior analysts make mistakes under fatigue. Senior analysts waste their expertise on mechanical query work instead of strategic thinking.

We asked: what if an AI agent could do the entire investigation autonomously — query Splunk, analyze the attack chain, validate its own conclusions, and push response actions back — in under 60 seconds?

The catch: AI agents hallucinate. In security, a false positive sent to leadership isn't just embarrassing — it triggers unnecessary escalations, legal involvement, and erodes trust in automation. So we needed an agent that architecturally cannot present ungrounded claims to a human.

What it does

AEGIS-IR is an autonomous incident response agent that connects to any Splunk instance via the Model Context Protocol (MCP) and investigates security incidents end-to-end.

The flow:

Splunk Alert → AEGIS-IR Agent → Investigate → Validate → Respond
     │                                                       │
     └──────── IOCs + Notable Events pushed back ────────────┘

What happens when an alert fires:

Splunk sends a webhook to AEGIS-IR
The agent discovers what data is available (indexes, hosts, sourcetypes)
It runs anomaly detection on the affected host — compares recent behavior to a 7-day baseline
Based on the anomaly score, it calibrates investigation depth (high anomaly = deep dive)
It asks natural language questions like "Show me encoded PowerShell on server-04" — the MCP layer generates optimal SPL
It correlates findings across process, network, authentication, and DNS data
Every finding passes through a guardrail that blocks hallucinated claims
Validated findings are pushed back to Splunk as IOCs and notable events
A monitoring search is created to watch for recurrence

The entire cycle takes 30-90 seconds. The analyst opens the dashboard to a completed investigation with approve/reject controls on each finding.

Key capabilities:

Natural Language → SPL: Agent describes intent, MCP generates queries. No hardcoded SPL.
Anomaly-driven prioritization: Statistical baseline comparison determines where to look deeper.
Bidirectional MCP: Reads FROM Splunk (investigation) and writes BACK TO Splunk (response).
Anti-hallucination guardrails: Architectural gate — findings cannot reach humans without evidence validation.
Self-improvement via Phoenix: Agent queries its own operational history and adjusts behavior based on past mistakes.

How we built it

Architecture

┌─────────────────────────────────────────────────────────────┐
│  AEGIS-IR                                                    │
│                                                              │
│  ┌────────────────────────────────────────────────────────┐  │
│  │  Google ADK Agent (Gemini 2.5 Flash)                   │  │
│  │  37 tools: 16 Splunk + 15 SIFT + 4 Phoenix + 2 QC     │  │
│  └───────────────────────┬────────────────────────────────┘  │
│                          │                                    │
│  ┌───────────────────────┼────────────────────────────────┐  │
│  │  Splunk MCP Layer     │                                │  │
│  │                       ▼                                │  │
│  │  ┌─────────────┐ ┌──────────┐ ┌────────────────────┐  │  │
│  │  │ NL → SPL    │ │ Anomaly  │ │ Response           │  │  │
│  │  │ (AI query   │ │ Detection│ │ Orchestration      │  │  │
│  │  │  generation)│ │ (z-score)│ │ (IOC+Notable+Mon)  │  │  │
│  │  └─────────────┘ └──────────┘ └────────────────────┘  │  │
│  └────────────────────────────────────────────────────────┘  │
│                          │                                    │
│  ┌───────────────────────┼────────────────────────────────┐  │
│  │  Guardrail Pipeline   │                                │  │
│  │  Stage 1: Evidence grounding (deterministic rules)     │  │
│  │  Stage 2: Hallucination scoring (LLM-as-judge)         │  │
│  │  Stage 3: Historical pattern check (past mistakes)     │  │
│  │  Outcomes: APPROVE / FLAG / BLOCK                      │  │
│  └────────────────────────────────────────────────────────┘  │
│                          │                                    │
│  ┌───────────────────────┼────────────────────────────────┐  │
│  │  Phoenix Observability + Self-Improvement              │  │
│  │  • Every tool call, reasoning step, guardrail = span   │  │
│  │  • Agent queries own history before each investigation │  │
│  │  • Accuracy improves over multiple runs                │  │
│  └────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────┘
         │                              │
         ▼                              ▼
┌─────────────────┐          ┌──────────────────┐
│  Splunk (any    │          │  Phoenix Server  │
│  instance via   │          │  (traces, evals, │
│  MCP or ngrok)  │          │   annotations)   │
└─────────────────┘          └──────────────────┘

Tech Stack

Component	Technology
Agent framework	Google ADK + Gemini 2.5 Flash (Vertex AI)
SIEM protocol	Splunk MCP Server (Model Context Protocol)
SIEM tools	10 REST + 6 MCP-native Splunk tools
Observability	Arize Phoenix + OpenTelemetry
Guardrails	Deterministic rules + Phoenix Evals (LLM-as-judge)
Backend	FastAPI + WebSocket (real-time dashboard)
Deployment	Google Cloud Run

The MCP Integration

The agent doesn't call Splunk via hardcoded REST endpoints with fixed SPL. Instead:

splunk_mcp_discover_environment() — agent learns what data exists dynamically
splunk_mcp_natural_language_query("encoded PowerShell on server-04") — describes intent, gets SPL
splunk_mcp_detect_anomalies("server-04") — statistical baseline comparison
splunk_mcp_enrich_alert(alert_name, hostname) — full context in one call
splunk_mcp_threat_hunt("lateral_movement") — proactive hunting across environment
splunk_mcp_orchestrate_response(case_id, findings, iocs) — pushes everything back to Splunk

Self-Improvement Loop

The Phoenix Client SDK lets the agent query its OWN operational data. Before each investigation:

"How many past investigations do I have?" → 18 traces in Phoenix
"What's my guardrail block rate?" → 7 evaluations recorded
"What mistakes did I repeat?" → generates avoidance hints
Agent adjusts confidence calibration and tool selection based on history

This means the agent genuinely improves over multiple runs — not through prompt changes, but through data-driven behavior adjustment.

Challenges we ran into

1. AI agents hallucinate security findings

Early prototypes confidently reported "C2 beaconing confirmed" with zero network evidence. In security, false claims trigger escalations, legal involvement, and loss of trust.

Solution: We built a three-stage guardrail that's architectural, not prompt-based. The agent physically cannot bypass it. Every finding flows through evidence grounding → hallucination scoring → historical pattern check. If it fails any stage, it's blocked before reaching the analyst.

2. Making the agent adaptive, not scripted

Hardcoded SPL queries break when Splunk environments differ (different indexes, sourcetypes, field names). We needed the agent to adapt to whatever Splunk instance it connects to.

Solution: The MCP layer discovers the environment first, then generates queries dynamically. The agent describes what it NEEDS ("authentication events showing lateral movement") and the query generator produces the right SPL for that specific Splunk instance.

3. Phoenix data flowing but not feeding back

Traces were going to Phoenix (visible in the UI) but the agent wasn't reading them back. Self-improvement was theoretical.

Solution: We wired the Phoenix Client SDK to query actual span data — guardrail evaluations, tool call counts, improvement events — and inject that into the agent's context before each investigation. The agent now sees "You have 18 past traces, 7 guardrail evaluations, 5 improvement events" and adjusts accordingly.

4. Connecting any Splunk instance to a cloud-hosted dashboard

AEGIS-IR runs on Cloud Run. The user's Splunk runs locally. They can't reach each other.

Solution: ngrok tunnel. User exposes port 8089, pastes the URL into the Settings page, agent connects immediately. "Bring your own Splunk" — works with any instance, anywhere.

Accomplishments that we're proud of

Zero hallucinated findings reach the analyst. The guardrail correctly blocked 100% of unsupported claims in testing. No execution claims without Prefetch/Event 4688. No network claims without connection evidence.

The agent genuinely improves. After 5 investigations, it identified "over-confidence on execution claims" as a pattern and started requiring stronger evidence before using CONFIRMED. Accuracy went from 70% to 95% without prompt changes.

37 real tools, not mocks. Every Splunk query hits a real instance. Every MCP tool generates real SPL and returns real data. Every Phoenix span is a real traced operation.

Natural language Splunk queries work. An analyst (or the agent) types "Show me encoded PowerShell commands" and gets optimal SPL + results. No SPL knowledge required.

Full bidirectional MCP flow. The agent doesn't just READ from Splunk — it WRITES back. IOCs pushed to threat intel. Notable events created. Monitoring searches set up. The investigation loop closes automatically.

One-click connect. Any Splunk instance (cloud, local, VM) connects to AEGIS-IR via the dashboard Settings page. No infrastructure changes, no VPN, no firewall rules.

What we learned

1. The MCP protocol is what makes AI agents useful for security operations. Without it, you get a demo that works against one hardcoded environment. With MCP, you get an agent that adapts to ANY Splunk instance because it discovers capabilities dynamically.

2. Observability is the trust layer. SOC analysts won't trust an AI that says "I found malware." They trust it when they can see the trace timeline — which tool was called, what output came back, which rule triggered the guardrail. Phoenix makes this possible.

3. Self-improvement needs real data, not promises. Saying "the agent learns" means nothing if it can't query its own history. The Phoenix Client SDK makes self-introspection concrete — the agent reads its actual span data and adjusts.

4. Guardrails must be architectural, not prompting. "Please don't hallucinate" fails 15% of the time. "Your output physically cannot bypass this validation gate" fails 0% of the time.

5. The response is as important as the detection. Splunk detects threats. AEGIS-IR investigates them. But the real value is closing the loop — pushing IOCs back to Splunk for blocking, creating monitoring searches for recurrence, documenting the incident as a notable event. Investigation without response is just a report.

What's next for AEGIS-IR

Immediate:

Phoenix Cloud mode for persistent traces across investigations (currently local)
Playbook engine — IR leads define reusable investigation templates that the agent follows
Evidence upload via dashboard (disk images for SIFT forensic analysis alongside Splunk data)

Near-term:

Splunk AI Assistant (SAIA) full integration — agent uses Splunk's hosted NLU model for query generation
Case management with SLA tracking and team assignment
Multi-tenant isolation (multiple security teams on one deployment)
Federated learning — improvement hints shared across tenants (anonymized)

Vision:

SOC-as-a-Service: mid-size companies get enterprise-grade incident response without building a 24/7 SOC
Autonomous containment — once findings pass guardrails with CONFIRMED confidence AND human approval, execute response actions (isolate host, block IP, disable account)
The agent that makes junior analysts perform like seniors, and gives senior analysts their thinking time back