Inspiration

Security teams are drowning. A mid-sized enterprise SOC receives over 10,000 alerts every single day — and 80% of them are false positives. Analysts spend their shifts chasing noise while real attacks slip through. When an actual incident is found, junior analysts miss critical forensic steps because institutional knowledge lives in senior analysts' heads, not in systems. And when a good analyst leaves, that knowledge walks out the door with them. We wanted to build something that doesn't just automate the investigation — it gets smarter after every single one.

What It Does

PhantomSOC is an autonomous incident response platform with a self-improvement loop at its core. Drop any security alert into it and three agents go to work: Layer 1 — SOC Triage Agent reads your detection playbook, queries its memory of past investigations, and makes a decision: false positive or escalate. No human needed for Tier-1. Layer 2 — Phantom Forensic Agent takes escalated alerts and runs a full DFIR investigation — building an attack timeline, mapping MITRE ATT&CK tactics, calculating a breach risk score with financial exposure, generating a GDPR 72-hour notification flag, and producing four stakeholder reports simultaneously: technical for the SOC analyst, risk summary for the security manager, business impact for the executive, and compliance obligations for legal. Layer 3 — Learning Meta-Agent is what makes PhantomSOC different. After every investigation, an LLM Judge scores the quality of both agents (0–100%). Those scores — along with confidence drift metrics — are traced to Arize Phoenix. The Learning Agent then queries its own Phoenix traces via MCP, identifies recurring blind spots and patterns of overconfidence, and automatically rewrites the DFIR and SOC playbooks. The next investigation runs on better rules.

How We Built It

The entire platform runs on Google ADK 2.1.0 as the agent runtime — chosen specifically because the Arize track requires a code-owned runtime for direct OpenInference instrumentation, which Agent Builder alone doesn't support. Every one of the six Gemini 3.1 Flash-Lite calls per investigation is automatically traced via the OpenInference google-genai instrumentor. Those traces land in Arize Phoenix Cloud with custom span attributes: judge scores, drift values, breach risk levels. The Learning Agent then calls the Phoenix MCP server at runtime to query its own historical trace data and close the loop. Investigation memory is stored in SQLite for cross-case IOC correlation. Executive reports and updated playbooks are persisted to Google Cloud Storage. The whole thing is containerized and deployed on Cloud Run with a public API and a judge-facing dashboard.

Challenges We Ran Into

Getting the self-improvement loop to actually close was harder than it looked on paper. The Learning Agent needs to query Phoenix MCP, extract meaningful patterns from trace data, and produce playbook rewrites that are structured enough for the next agent to use — all in a single pipeline run. Early versions produced vague feedback that didn't translate into actionable playbook changes. Confidence drift detection required careful calibration — the gap between an agent's self-reported confidence and the LLM Judge's external score had to be measured consistently across different alert types to avoid false drift alarms. Wiring OpenInference instrumentation into ADK's code-owned runtime without losing span context across agent boundaries also took significant iteration.

Accomplishments That We're Proud Of

The before/after learning results came out better than expected. DFIR quality jumped from 58% to 77% after one learning cycle. SOC quality went from 50% to 75%. MITRE tactic coverage doubled from 3 to 6. Confidence drift went from CRITICAL to WARNING. The Phoenix MCP integration — an agent querying its own observability data to rewrite its own playbooks — is genuinely novel. Most AI SOC tools use Phoenix as a passive logger. PhantomSOC uses it as an active data source that drives system behavior.

What We Learned

LLM observability is only valuable if something acts on it. The industry has been building better and better dashboards for model behavior — but dashboards require humans to read them and translate insights into action. Closing that loop autonomously is where the real leverage is. We also learned that self-improvement is fragile without a good judge. The quality of the Learning Agent's playbook rewrites is entirely dependent on how well the LLM Judge scores investigations. Getting that scoring prompt right was the highest-leverage work in the entire project.

What's Next for PhantomSOC

Real-time SIEM integration — connecting directly to Splunk, Elastic SIEM, or Microsoft Sentinel so alerts flow in automatically rather than through manual API calls. Multi-tenant memory — separate investigation histories per client, making PhantomSOC viable for MSSPs managing dozens of organizations simultaneously. Adversarial playbook testing — using the Learning Agent to simulate attack variations and proactively identify gaps in detection rules before real incidents expose them. And eventually, a human-in-the-loop escalation layer — where PhantomSOC handles everything it's confident about autonomously and surfaces only genuine ambiguity to a human analyst with full context already assembled.

Built With

• arize-phoenix
• gemini
• google-adk
• google-cloud
• google-cloud-run
• openinference
• python
• sqlite