Inspiration
Traditional infrastructure monitoring operates on a reactive paradigm: metrics cross a critical threshold, an alert is dispatched, and human analysts scramble to diagnose and patch the system. In the context of sovereign, decentralized validator networks running high-throughput consensus mechanisms like Proof of Karma, this human latency is unacceptable. A single minute of down-time or a compromised kernel context can degrade network consensus and compromise ledger integrity.
We were inspired to build DharmaGate: Agentic Fleet Resilience to eliminate the monitoring gap entirely. Our vision was to shift from passive, reactive observibility to Autonomous Operational Immunity—creating an infrastructure stack that utilizes LLMs not as simple chatbots, but as type-safe, deterministic systems engineering brains capable of inspecting, defending, and optimizing production environments at machine speed.
How we built it
DharmaGate is architected as a closed-loop feedback system that bridges low-level kernel telemetry with agentic strategic reasoning.
- The Telemetry Pipeline: Production validator nodes within our Mega-Compose network run automated eBPF (Extended Berkeley Packet Filter) and XDP sensors directly inside an Ubuntu LTS environment. This low-level telemetry streams live consensus hearbeats, microsecond network latencies, and system-call patterns into Splunk Enterprise via localized Splunk Universal Forwarders.
- The Agentic Core: We implemented a custom Model Context Protocol (MCP) Server for Splunk. This acts as a secure, type-safe abstraction layer that allows our autonomous AI Agent to query live log indexes, perform statistical analysis, and cross-correlate alerts using strict, deterministic function calls.
- Cryptographic Auditing: To prevent the agent from becoming a "black box," we built an internal Chain-Hash Auditor. Every reasoning loop, query execution, and mitigation intent is cryptographically bound to the hash of the preceding event:
$$\text{Hash}n = \text{SHA-256}(\text{Timestamp} \parallel \text{Intent} \parallel \text{Action} \parallel \text{Status} \parallel \text{Hash}{n-1})$$
This structured JSON trail is continuously streamed back into an immutable Splunk index (
audit_index), ensuring an unforgeable ledger of autonomous decisions.
Challenges we faced
- The Determinism Trade-off: Giving an LLM direct access to infrastructure primitives (like modifying container resources or dropping network packets) introduces massive security and operational risks. We solved this by strictly typing the agent's action capabilities via the Model Context Protocol, confining its operational boundary to precise, parameterized tools rather than raw shell execution.
- Telemetry Naming Synchronization: Moving from simulated hackathon datasets to production telemetry revealed massive naming mismatches between real-world infrastructure abstractions (
validator-{1..7}.radhikachain.internal) and early codebase mockups. We had to rewrite our log parsing pipelines to guarantee that the agent could accurately resolve real-world container identities under adversarial pressure. - Network Visibility Without Exposure: Running a public-facing analytical backend on a cloud instance usually requires opening insecure ingress ports. We bypassed this entirely by containerizing a Cloudflare Tunnel (
cloudflared) inside our deployment. This allowed our Splunk Enterprise interface and HEC endpoints to remain completely hidden behind Cloudflare Zero Trust, providing a secure, encrypted URI for our frontend without exposing a single open port to the public internet.
What we learned
We discovered that the intersection of low-level kernel observation (eBPF) and high-level abstract reasoning (Agentic LLMs) is where true infrastructure resilience lives. We learned how to balance complex data indexing pipelines in Splunk with the concise token context windows required by frontier models. Most importantly, we proved that cryptographic chain-hashing can bring absolute auditability and accountability to autonomous AI operators, making code sovereignty and security complementary rather than conflicting forces.
Log in or sign up for Devpost to join the conversation.