AgentGuard

Inspiration

Multi-agent AI systems are moving from demos to production, but when a chain of LLM calls, tools, and sub-agents fails at 2 a.m., teams still grep logs and guess. Many enterprises already run Splunk for security and ops; they should not need a second observability stack just for agents.

That led to a design where I added two Python decorators, spans land in Splunk via HEC, and on-call engineers (or Claude via MCP) can ask: “Why did research_agent fail for trace X?” This inspired AgentGuard.

How I Built It

AgentGuard is an enterprise-grade observability platform designed explicitly for multi-agent systems. The architecture is split into two distinct, real-time workflows:

• Telemetry Ingestion: I built a lightweight Python SDK that hooks into multi-agent loops, capturing high-consistency execution traces, state changes, and host-level metrics. Whenever an agent panics or hits a resource threshold, the SDK packages the payload as JSON and securely POSTs it to Splunk Enterprise via the HTTP Event Collector (HEC).
• AI Diagnostics: To bring reasoning directly to the operational data layer, I deployed the native Splunk MCP Server. This HTTP-based bridge connects the local Splunk instance to Claude Desktop via the Model Context Protocol. By exposing the live Splunk main index as a suite of callable tools, Claude dynamically executes native SPL queries at runtime to isolate root causes—such as autonomously deducing that an agent's RuntimeError was caused by creeping 85.5% memory exhaustion rather than a CPU spike.

Challenges I Faced

Bridging the gap between a local LLM agent and secure infrastructure presented several distinct hurdles. The most significant challenge was local networking and security policies. Because the Splunk MCP server operates over an HTTP transport using Node.js (npx mcp-remote), Node's strict built-in security aggressively rejected Splunk's default self-signed SSL certificates with SELF_SIGNED_CERT_IN_CHAIN errors. I had to carefully modify the environment configuration to bypass the TLS unauthorized rejection without breaking the connection payload. Additionally, ensuring that highly asynchronous, multi-agent telemetry remained strictly ordered and properly formatted for Splunk's HEC required precise payload structuring.

What I Learned

This project solidified my belief in the Model Context Protocol as the future of infrastructure management. I learned how to seamlessly translate raw, unstructured multi-agent execution states into high-consistency JSON for Splunk ingestion. More importantly, I saw firsthand how pairing structured operational data with an LLM's runtime reasoning shifts the industry from reactive monitoring to proactive, self-diagnosing systems.

Built With

• django
• hec
• mcp
• python
• rest
• sdk
• splunk