gemini-splunk-devx-agent

A Splunk admin copilot on the Splunk MCP. Walks list_apps / list_savedsearches / get_savedsearch / list_kvstore_collections / audit_knowledge_objects and emits a ranked cleanup punch-list.

Comment

Why this exists

When a Splunk admin inherits a tenant, the manual chore is reading every drawer in a stranger's filing cabinet. Open Settings → Apps. Click each one. Open Settings → Searches and click through every savedsearch's schedule, owner, and SPL. Check the KV store. Check the lookup directory. Cross-reference owners against the offboarded-users list. By the time you have a punch-list, half a day is gone.

gemini-splunk-devx-agent does that walk for you. Hand it the tenant and it produces a 7-section audit report that quotes every issue verbatim from the Splunk Platform MCP tool output.

What it does

The system prompt forces a five-step workflow over the Splunk Platform MCP tool surface:

  1. list_apps(include_disabled=True) — get the full app inventory + versions + disabled state.
  2. list_savedsearches(app=None) — enumerate every scheduled / ad-hoc savedsearch.
  3. get_savedsearch(name) — for any suspicious savedsearch (lives in a disabled app, deprecated SPL command, offboarded owner), pull the full record + per-object lint issues.
  4. list_kvstore_collections() — KV inventory with record counts, sizes, and last-modified timestamps.
  5. audit_knowledge_objects() — lint pass that returns issues grouped by severity (high / medium / low) plus a ranked cleanup punch-list.

The agent emits EXACTLY these labeled sections:

INVENTORY:           one sentence with the app / savedsearch / KV counts.
ENVIRONMENT:         2-4 bullets — notable apps / disabled apps / heavy owners.
HIGH-SEVERITY ISSUES:    verbatim audit_knowledge_objects entries (severity=high).
MEDIUM-SEVERITY ISSUES:  same, medium.
LOW-SEVERITY ISSUES:     same, low.
CLEANUP PUNCH-LIST:  top 5 ranked actions, most-impactful first.
NEXT STEP:           one concrete first action.

The differentiator is the CLEANUP PUNCH-LIST: the agent doesn't just dump the audit. It ranks issues by real operational impact (a deprecated SPL command on a still-scheduled search owned by an offboarded user beats a stale KV store nobody references).

The canned inherited tenant

Six apps, one of them (TA-deprecated-2024) disabled. Five savedsearches across the apps, including one that's actively scheduled inside the disabled app — using the removed sendresults command, owned by ex_admin_a (offboarded). Five KV store collections, two of them idle 180-412 days. Two orphaned lookups (4,817 and 12,900 rows respectively) in the deprecated app.

audit_knowledge_objects returns 8 grouped issues (2 high + 3 medium + 2 low + the in-disabled-app finding). The agent's punch-list ranks them by impact: kill the deprecated search first because it WILL fail on its next 30-minute fire and the failure will cascade.

Live Vertex AI smoke test

Reproducible via scripts/smoke.py. The latest live run passed all eight verbatim-output checks:

[PASS] has INVENTORY section
[PASS] has ENVIRONMENT section
[PASS] has HIGH-SEVERITY section
[PASS] has CLEANUP PUNCH-LIST section
[PASS] has NEXT STEP section
[PASS] names deprecated TA verbatim
[PASS] flags sendresults command verbatim
[PASS] names a KV collection verbatim

The agent's actual output names TA-deprecated-2024, quotes the sendresults deprecation, calls out ex_admin_a as offboarded, and ranks the punch-list with the deprecated savedsearch as item #1.

How we built it

  • Model: Gemini 2.5 Flash on Vertex AI.
  • Agent runtime: google.adk.agents.LlmAgent + McpToolset (Google Cloud Agent Builder / ADK).
  • MCP: Splunk Platform tool surface — bundled stub for demos, real-tenant-ready via env vars (SPLUNK_HOST, SPLUNK_TOKEN).
  • Surface: Streamlit dashboard on Cloud Run + runner.ask() Python entrypoint.
  • License: Apache 2.0, standalone repo created during the contest period.

Bonus prizes targeted

  • Best Use of Splunk MCP Server ($1K) — the agent's entire tool surface IS the Splunk MCP server. Five tools (list_apps, list_savedsearches, get_savedsearch, list_kvstore_collections, audit_knowledge_objects) match the official server's shape, stubbed for demos and real-tenant-ready via one env-var swap.

Why this is substantially different from gemini-splunk-agent and gemini-splunk-security-agent

This is my third submission to Splunk Agentic Ops:

  • gemini-splunk-agent (Observability track) — 5 tools for SRE incident investigation: list_alerts, get_detector, list_indexes, run_search, run_observability_query.
  • gemini-splunk-security-agent (Security track) — 5 tools for SOC notable-event triage: list_notable_events, get_notable_event, threat_intel_lookup, asset_lookup, get_soar_playbook_state.
  • gemini-splunk-devx-agent (THIS submission — Platform & Developer Experience track) — 5 tools for admin knowledge-object audit: list_apps, list_savedsearches, get_savedsearch, list_kvstore_collections, audit_knowledge_objects.

Different track, different tool surface, different user persona (admin vs SRE vs SOC), different output shape (audit + punch-list vs root-cause verdict vs self-correction verdict).

License

Apache 2.0. Standalone repo created during the Splunk Agentic Ops contest period.

Built With

  • agent-development-kit
  • apache-2
  • gemini
  • gemini-2-5
  • google-cloud-agent-builder
  • google-cloud-run
  • mcp
  • model-context-protocol
  • python
  • splunk
  • splunk-knowledge-objects
  • splunk-mcp
  • splunk-platform
  • streamlit
  • vertex-ai
Share this project:

Updates