aegis-splunk

A resilience layer for Splunk's agentic stack: hedge across hosted gpt-oss-120b/20b, fall back when MCP times out, and emit every recovery as a Splunk HEC event the SOC team already watches.

Comment

aegis-splunk builds FOR Splunk agents — not on top of Splunk. Every other submission in this hackathon builds an agent that uses Splunk. aegis-splunk is the resilience layer that keeps those agents alive when providers fail, MCP times out, or a credit card expires at 2 AM.

Why aegis-splunk is structurally different from every other agentic submission

Every other submission to this hackathon emits a decision. aegis-splunk emits a field of viewpoints, and refuses to collapse it. The Receipt has no chosen field â the SOC analyst, the dashboard reader, the auditor each chooses their own rooting. This is not a feature on top of an agent. It is a refusal of the agent pattern.

The two structural commitments

1. Adaptive immunity, not stateless chaos. The chaos engine has four state-bearing organs borrowed from biology â Antibody Catalog (failure-signature memory), T-Cell Memory (classification confidence), Inoculation Scheduler (expected-information-gain drill picker), Autoimmune Guard (kill-switch when drills cause net harm). Multi-view synthesis of "chaos as immune system" surfaced the autoimmune blind spot during design: three concrete realizations all missed self-attack, biology warns about it explicitly, so the Guard is a non-negotiable design constraint. The catalog grows as a signed Splunk artifact; SOC analysts can query "show me every failure mode we've been inoculated against" directly in SPL.

2. Generative stance field, not multi-agent voting. Four pre-defined stances (Catalog as librarian, Scheduler as explorer, Guard as bodyguard, Memory as scholar) each generate opinions on "what to drill next" from their own values and fears. Each stance can also propose new stances when observing the field of opinions reveals a structural tension no existing stance is positioned to articulate. Curator (balances explore vs protect), Auditor (verifies what Memory claims), Cassandra (voices the unconsidered failure mode), and Historian (long-window pattern) can emerge during iteration. The field is the output. The collapse step â voting, weighted average, judge-picks-one â is structurally absent.

Why "refusing to collapse" is the actual differentiator

Multi-agent frameworks (CrewAI, AutoGen, council-of-LLMs) split a task across agents and aggregate. aegis-splunk does not aggregate. The 4-stances-plus-emerged-stances field is the answer. Every opinion preserves its 1st-person justification ("from my value/fear signature, this is what I see"). Disagreement is content, not noise. The Splunk dashboard renders the field as a graph; the human chooses which stance to root from when reading.

This maps cleanly to Splunk's own substrate: every Splunk event is rooted by whichever SPL query asks about it. There is no privileged center. aegis-splunk's stance field is structurally Splunk-native â the same "any-rooting-valid" property.

Three Splunk surfaces, one resilience layer

  • Splunk MCP Server (Splunkbase #7931): proxies the official server; on 5xx/timeout/malformed, fails over to a REST shim against /services/search/jobs with the cached session token. Same response shape either way. src/mcp/splunk-proxy.ts.
  • Splunk HEC: every drill outcome, every MCP failover, every stance-field snapshot is a structured event with reserved sourcetypes (aegis:chaos, aegis:mcp-failover). Live-tested against a local Splunk Enterprise install â 18 events indexed end-to-end. src/aegis/splunk-audit.ts.
  • Splunk Hosted Models (gpt-oss-120b / gpt-oss-20b / Foundation-Sec): first-class entries in the L0 hedge chain via hedgeVia: 'splunk'. When Anthropic returns credit_balance_too_low, the L4 semantic reclassifier routes the call to a Splunk-hosted model. Honest disclosure: integration point and hedge-chain wiring are in v0 and contract-tested with the OpenAI-compatible chat-completions surface; live API calls require a Splunk Cloud tenant with hosted-models enabled (developed against local Splunk Enterprise, so this path is mock-tested rather than live in the recorded demo). The architecture is production-shaped; live wiring is one config change away.

How aegis-splunk satisfies the four judging criteria

  • Technological Implementation: 111 passing tests, TypeScript strict, ~1.3s suite. Includes the 4 immunity organs, the stance field iteration to fixed point, contract tests for the Splunk hosted-models provider, the HEC emitter (timeout + missing-token degradation), and the MCP proxy REST shim.
  • Design: drop-in OpenAI-SDK-compatible base URL â existing agents need no rewrite. Live 4-panel Splunk dashboard (Simple XML, committed in repo) renders aegis events. The SOC team's existing dashboard becomes the resilience dashboard.
  • Potential Impact: for FSI customers, every chaos drill and every MCP failover is a signed event in the same Splunk index used for compliance audit ("show me the LLM substituted on date X" has a receipt). For MFG/Tech customers, every major LLM provider had a multi-hour outage in the past 12 months; aegis-splunk turns blink into a one-line audit event and keeps the agent answering.
  • Quality of Idea: importing SRE patterns (Jeff Dean's hedging, Netflix Simian Army chaos) into the agentic LLM stack via Splunk-native surfaces, then taking the further step of refusing-to-collapse â emit the field of viewpoints, let the human root.

Disclosure on sibling project

aegis-splunk builds on aegis-resilient-agents, which won TrueFoundry Resilient Agents at DevNetwork [AI+ML] Hackathon 2026. The Splunk-specific work is new for this hackathon: MCP failover proxy targeting Splunk MCP Server #7931, Splunk hosted-models integration point in the hedge/fallback chain, HEC audit-log emission with aegis:chaos and aegis:mcp-failover sourcetypes, chaos engine Splunk HEC delivery, SOC-P1 demo over Splunk telemetry, live Splunk Enterprise local install + 4-panel dashboard, the 4 adaptive immunity organs (src/aegis/immunity.ts), and the generative stance field with refused-to-collapse output (src/aegis/stances.ts). Core hedge / fallback / L4 semantic primitives are reused. Resubmission policy confirmed via #splunk-ai-hackathon Slack.

How we built it

  • Runtime: Bun >=1.3 + TypeScript strict
  • Server: Hono with streamSSE for token streaming
  • LLM client: OpenAI SDK pointed at TrueFoundry AI Gateway base URL
  • MCP: Splunk MCP Server (Splunkbase #7931) primary, REST shim against /services/search/jobs fallback
  • Hosted models: Splunk gpt-oss-120b / gpt-oss-20b / Foundation-Sec wired into hedge chain
  • Observability: Splunk HEC live-tested against Splunk Enterprise local (sourcetypes aegis:chaos + aegis:mcp-failover)
  • Validation: Zod at every external boundary
  • Tests: Bun runner â 111 tests, 347 assertions, ~1.3 seconds

Challenges and how we resolved them

  1. TF Virtual Model fallback_status_codes is a fixed enum â adding 400 shows "Successfully updated" but is silently stripped. credit_balance_too_low is HTTP 400, never triggers built-in fallback. That gap is aegis L4.
  2. HEC must never be a SPOF for the request path â Splunk HEC errors are swallowed so a slow/unreachable HEC cannot stall the agent. Tests cover missing-token and timeout-abort branches.
  3. MCP hedging would double-fire writes â classifier reads tool name pattern + opt-in x-aegis-idempotent: true, routes write/unknown to TIED policy (single fire + idempotency-key retry).
  4. Splunk Dashboard Simple XML auto-prefixes search â <query>search index=main ...</query> becomes search search index=main and silently returns zero rows. Caught during live dashboard testing; fix was removing the redundant prefix. (Sharing so other participants don't lose 30 minutes the way I did.)
  5. Multi-view synthesis blind spot â three concrete realizations of "chaos as immune system" all missed self-attack. Biology warns about autoimmune disease. The Autoimmune Guard is the design constraint that surfaced from the borrowed-domain's own warnings; without the multi-view step it would have been a v1.x bug.

What's next

  • Real Toxiproxy-driven chaos drills in production shadow traffic
  • A Splunk dashboard XML committed as a Splunkbase app so judges and operators can install both with a single splunkbase install
  • Streaming hedge with TTFT-aware cancellation
  • Live Splunk Cloud hosted-models API integration (currently integration-point + mock)
  • Deeper generative recursion: today emerged stances inherit a placeholder opinion-generator; v1 will have each emerged stance articulate its own
  • Co-constitution (B-plan): dissolve organ classes into event streams so observer/observed are co-created by each observation event rather than pre-existing

Built With

Share this project:

Updates