splunk_obs + sfos-evolver: closed-loop agent ops

• 

  The silent regression caught in one frame: translate-en-to-zh at 1.13 vs cohort 1.87-3.40

• 

  Panel 2b - Low-tail p10 drift alert. translate-en-to-zh outlier at 1.13/5 score across 3 runs

• 

  Panel 2 - Mean Brier eval score across 6 skills over 30 days. Silent regression becomes visible

• 

  evolver.py:223 find_drift_patterns - reads drift signal, asks Haiku for a patch inside a hard whitelist

Inspiration

I run 11 AI agents on cron at $0.06/week. In May, my translate-en-to-zh skill silently dropped from 4.0 to 1.13 out of 5. Nothing crashed. Nothing paged. No exception. The agent kept generating Chinese translations, they just got worse every day. I caught it 9 days later, by hand, looking at JSON files. That is the gap Splunk Agentic Ops should close: not "did the agent run" but "is the agent still good".

What it does

Two parts, one loop:

1. splunk_obs is a 200-line stdlib-only HEC adapter (no requests, no httpx, no Splunk SDK). It tails three streams into Splunk: Reflexion outcomes (sfos:reflection), eval drift (sfos:eval), Thompson bandit pulls (sfos:bandit). Ships a 6-panel dashboard.xml.

2. sfos-evolver is a weekly cron agent that reads the same drift signals from disk, finds patterns where the same failure repeated >=3 times or the same skill regressed >0.5 mean score across two runs, asks Claude Haiku to propose a code patch, and opens a GitHub PR. Hard whitelist: prompts / schemas / error-strings / markdown only. Never touches auth / network / money code.

The PR is the gate. No auto-merge. Human reviews and approves.

How we built it

Stdlib because enterprise pip install politics. HEC because every Splunk customer already has it open. dashboard.xml because Classic Dashboards work on every Splunk version since 2014. Evolver because Reflexion and DGM papers showed self-improvement works, but only when constrained, so I locked the whitelist before writing a single Haiku prompt. Eval scoring is Brier-style 1-to-5 from a Claude judge (sfos-eval). The whole loop, from Splunk dashboard panel to GitHub PR, lives in 8 Python files totaling 1,847 LOC with 132 tests. Solo Founder OS v0.28.0 on PyPI.

Challenges we ran into

The eval JSON schema drifted between the v0.23 emitter and the v0.28 translator (top-level mean vs nested scores[].overall). Caught it live during this hackathon when 14 events landed in Splunk all at score 0.0. 5-minute fix in sfos_translator.py. Real example of the exact failure mode the system is built to catch.

Accomplishments that we're proud of

• 6 panels rendering on real production data (11 agents x 30 days)
• Eval drift outlier visible at 1.13 / 5 (translate-en-to-zh) vs cohort 1.87 to 3.40
• Evolver wired to drift signals end-to-end (find_drift_patterns in evolver.py:223)
• Stdlib-only HEC client passes Splunk's 2014 dashboard format
• Whole stack runs on Splunk Enterprise Free 60-day license

What we learned

The hardest part of agent observability is not ingestion. Splunk solved that 15 years ago. It is defining what "good" means for a non-deterministic agent. Brier-style numeric eval per skill, scored weekly by a judge model, turned out to be the smallest unit that fits in a dashboard panel.

What's next for splunk_obs + sfos-evolver

1. EU AI Act Article 12 compliance recorder. Aug 2, 2026 binding date is 48 days out. 78% of orgs non-compliant. Article 12 mandates tamper-evident logs retained 6 to 24 months. splunk_obs is already that, packaged as "compliance recorder for solo operators" instead of "drift dashboard for hobbyists" widens the buyer 100x.
2. Publish the eval JSONL as a HuggingFace dataset (alexji/solo-founder-os-evals): 6 weeks of single-operator cron-judged traces, 34 rows across 7 skills, warts-and-all (PII-sanitized via regex heuristics). First public dataset of solo-founder agent eval traces. Karpathy "verification loops are the moat" thesis needs proof artifacts.
3. Wire splunk_obs to the Splunk MCP Server so the evolver can pull drift via MCP instead of disk (qualifies for the $1k MCP bonus track in v0.29).
4. Replace the Haiku judge with Splunk Hosted Models for SOC2-conscious users.

Why this matters now

Anthropic standardized Claude Skills as an open spec in December 2025. Andrej Karpathy named "create evaluation loops" as a core agentic-engineering job. splunk_obs + sfos-evolver + council-diff (sister OSS, persona-vs-persona Brier audit) form a Software 3.0 reference stack: observe -> judge -> patch.

Built With

• anthropic
• claude
• claude-haiku
• cron
• github-cli
• launchd
• mit-license
• pypi
• python
• splunk
• splunk-hec