DevLens

Inspiration

Every time a production application breaks, developers spend the first 20–40 minutes doing one thing: writing queries. Not debugging code, not fixing problems, just staring at a blank Splunk search bar trying to remember if it’s p99(response_time_ms) or perc99(latency).

While Splunk is incredibly powerful, most developers don’t speak fluent SPL. We realized there was a massive gap between the rich observability data available in Splunk and the engineers who actually need to use it to fix code. We were inspired by the concept of an "Agentic SOC" and wanted to bring that exact same autonomous intelligence to developers and observability.

What it does

DevLens is an autonomous AI observability agent built as a native Splunk Enterprise app. It allows developers to investigate application problems by asking plain-English questions, no SPL required.

When a developer asks "Why are my APIs returning 500s?", DevLens doesn't just act as a chatbot. It executes a Reason → Act → Observe agentic loop:

Plan: The Cisco Foundation AI determines the best SPL query to investigate the symptom. Act: It runs the query against live Splunk data via an MCP server integration. Observe: It analyzes the results. If it finds the root cause (e.g., Redis pool exhaustion), it synthesizes a final report. If not, it writes a deeper query and loops again. The final output gives the developer the root cause, a confidence score, exact evidence cited from their telemetry, actionable recommendations, and a generated SPL query they can save as an alert.

How we built it

We built DevLens as a full-stack native Splunk app in just a few days:

The Intelligence (Agentic Loop): We built a Python orchestrator (agent.py) that manages the multi-step investigation loop, tightly integrating with Splunk Hosted Models (Cisco Foundation AI) for reasoning and anomaly detection. The Tooling (MCP Server): We registered 5 distinct MCP tools (devlens_investigate, devlens_spl_generate, etc.) inside tools.conf, making DevLens's capabilities accessible not just internally, but to any external MCP-compatible AI agent. The UI: We designed a dark-mode, terminal-inspired Splunk Simple XML dashboard paired with custom JavaScript and CSS to render the AI's "thinking" steps dynamically. The Data: We generated over 1 million rows of synthetic telemetry data (HTTP access logs, infrastructure metrics, deployment events) containing hidden edge-case regressions to prove the agent's effectiveness.

Challenges we ran into

Building an agentic loop that actually executes queries against live data was risky. Our biggest challenge was preventing the LLM from hallucinating invalid SPL syntax that would crash the investigation. We overcame this by building a hybrid spl_generator.py module that combines curated, high-efficiency SPL templates with the LLM's dynamic filtering, ensuring every query executed against the Splunk REST API was syntactically valid and performant.

Additionally, navigating the Splunk app directory structures, ensuring proper file permissions for custom REST handlers, and rendering complex interactive HTML/JS inside a Simple XML dashboard required some deep dives into Splunk's architecture.

Accomplishments that we're proud of

e are incredibly proud of successfully implementing a true multi-step agentic loop inside Splunk. Unlike standard RAG implementations that just query a vector database, DevLens actively writes code (SPL), executes it, reads the data, and decides what to do next based on the numbers.

We are also proud of hitting our dual targets: successfully implementing the Model Context Protocol (MCP) via tools.conf and leveraging Splunk Hosted Models. We built a tool that genuinely lowers the barrier to entry for observability.

What we learned

e learned a tremendous amount about Splunk's internal architecture, specifically how to build and register custom REST API handlers (restmap.conf) and how to wire them securely to Python backend logic. We also learned how powerful the MCP standard is for creating modular, callable AI tools rather than monolithic applications.

What's next for DevLens

We want to take DevLens beyond just investigation and into remediation. In the future, DevLens will not only tell you that your Redis connection pool is exhausted, but it will automatically generate the Kubernetes PR to increase the REDIS_MAX_CONNECTIONS environment variable and ask for your approval to merge it. We also plan to integrate distributed tracing (OpenTelemetry) so the agent can trace 500 errors across microservice boundaries autonomously.

Built With

• css
• javascript
• model-context-protocol
• powershell
• python
• splunk-simple-xml
• splunk-spl