ZKSplunk: Project Story

The full narrative behind ZKSplunk, written for our Devpost submission and recorded here as part of the project's permanent history. Author: John M.P. Santi & Alex P., EnterpriseZK Labs LLC, June 2026

Inspiration

The Night Cardano Almost Split in Two: Why ZKSplunk Exists

The incident that started it

On November 21, 2025, the Cardano mainnet did something it had never done in eight years of operation: it split into two chains.

A single, deliberately crafted, malformed delegation transaction exploited a deserialization bug that had been sitting in a lower-level serialization library since 2022. Newer node versions accepted the transaction as valid. Older node versions correctly rejected it. From that one disagreement, the network diverged into two competing histories: a "poisoned" chain carrying the malformed transaction, and a "healthy" chain without it. Block producers kept building, each on whichever branch their software version believed in.

The fallout rippled outward fast. Exchanges paused ADA deposits and withdrawals. Block explorers froze or showed conflicting data. DeFi protocols hit a mismatched state, with interactions confirmed on one branch but not the other. ADA dropped more than 6%. No user funds were lost, but for several hours, one of the most rigorously engineered blockchains in existence was effectively partitioned.

The engineering response was genuinely impressive. Teams from Input Output Global (IOG), The Cardano Foundation, Intersect, and EMURGO coordinated an emergency response and shipped a patched node (10.5.3) in roughly three hours. Then came the slow part: every block-producing stake pool operator on Earth had to actually learn that a patch existed, understand why it mattered, download it, and restart their node, before the network could reconverge on a single canonical chain.

That last sentence is the whole reason ZKSplunk exists.

The gap Charles pointed at

In the aftermath, Charles Hoskinson was blunt about what the episode exposed. The fix was ready quickly. Getting it into the hands of a globally distributed, intentionally decentralized set of node operators was the hard part. There was no fast, reliable, fan-out channel to say to every operator at once: here is the anomaly, here is what it means, here is exactly what you need to do, right now.

His framing stuck with me. A publish and subscribe (pub-sub) coordination layer, where operators subscribe once and the network can broadcast a verified incident and remediation instruction to all of them simultaneously, would have let the ecosystem coordinate the workaround far faster and far better than scattered Discord messages, forum posts, and X threads.

Decentralization is a feature, not a bug. But decentralization without a coordination channel means that the moment something goes wrong, every operator is discovering the problem on their own clock. The chain reconverged in spite of that friction, not because the friction was solved. The detection-to-coordination loop was the weak link.

The moment Splunk clicked

I was already deep in the Midnight ecosystem, thinking about observability for zero-knowledge infrastructure, when the Cardano split happened. The two ideas collided.

What if the anomaly that fractured a chain could be seen the instant it appeared in the public ledger? Not after explorers froze. Not after someone noticed on social media. The instant the metrics deviated.

Then I started exploring Splunk, and found the piece that made me genuinely excited: Splunk can aggregate anomalies across a stream of telemetry and, when a pattern crosses a threshold, fire an alert action. One of those actions is email. You can detect a deviating signal and, in the same breath, notify the operations security admins who need to act on it. No human babysitting a dashboard at 2am. The pattern itself reaches out.

That is a pub-sub coordination layer hiding in plain sight, built on a platform enterprises already trust. The publisher is the anomaly detector watching the ledger. The subscribers are the opsec admins. The message is: here is the anomaly, here are the addresses involved, here is the action, here is what to do.

The thing Charles said was missing, Splunk can provide the operator-facing half of, and it can do it today.

What ZKSplunk does about it

ZKSplunk streams metrics from the Midnight public ledger and a DApp's own infrastructure into Splunk. When a nefarious pattern emerges — an anomalous outflow burst from an address, a flood of contract calls, a mint storm, a degraded proof server — Splunk aggregates the signal and can fire an alert action, including email, to the opsec admin. We ship the alert rules wired and a critical-event email rule ready to go; the operator points it at their SMTP, and the notification carries the charts and public data the rule is designed to surface: the addresses involved, the action observed, and a recommended course.

Observe, notify, decide. It is the operator-coordination layer the Cardano split showed the world it needed, applied to Midnight, where the chain is private by design but the public ledger and infrastructure still leave an observable surface worth watching.

We are not claiming we could have single-handedly prevented a chain split. We are saying the gap between something is wrong and every operator who needs to know, knows, with instructions should be measured in seconds, not hours. That gap is exactly what a detection plus pub-sub notification pipeline closes.

A known bug, a malformed transaction, two chains, and three hours of scrambling to tell the world how to fix it. We watched that happen, and we built the early-warning and coordination layer we wished had been there.

That is our inspiration.

What it does

ZKSplunk is the first Splunk connector for any zero-knowledge blockchain infrastructure on Earth (or any blockchain on Earth for that matter, to our knowledge). ZKSplunk takes Midnight, a chain that is private by design, where computation happens inside zero-knowledge proofs precisely so that no one can see inside, and gives operators a way to watch over it without ever breaking that privacy.

The whole system moves in three beats: observe, notify, decide.

It observes by streaming telemetry from two places, the Midnight public ledger and a DApp's own MidnightVitals console, into Splunk over the HTTP Event Collector. What makes this different from ordinary monitoring is that ZKSplunk actually understands the things that are unique to a ZK chain: proofs that normally take seventeen to twenty-eight seconds and can fail silently halfway through, proof servers that quietly run out of memory, indexers that drift out of sync, block cadence that stutters, and the fee-generation (DUST) health that keeps the whole machine paying its way.

It notifies. When a pattern crosses a threshold, Splunk fires an alert action. The app ships ten alert rules; most run scheduled and tracked in Splunk's Triggered Alerts so the demo works with no mail server, and a dedicated "Any Critical Event" email rule ships ready to deliver the moment an operator configures SMTP. This is the operator-facing half of the pub-sub coordination layer the Cardano split proved the world needed: the anomaly detector is the publisher, the operators are the subscribers, and nobody has to sit awake watching a dashboard at two in the morning.

And then it helps decide the course of action. A security layer we call zkZap re-reads that same telemetry as a stream of threat signals — proof floods, failed-authorization brute-force bursts, wallet drains, mint anomalies, and indexer outages — and surfaces them through detection-focused dashboards and an evidence-backed analyst loop that recommends a response. The analyst loop runs today; automated, hands-off response (SOAR-style) is the next step on the roadmap. The crucial discipline throughout is that zkZap only ever reads the public surface of the chain. On Midnight, metadata and volumes are public while contents stay private, so we built everything from what is observably public and never have access to the private state. That is not a limitation we apologize for; it is the line that makes the whole thing honest.

ZKSplunk does all of this through two lenses over a single pipeline. ZKSplunk Me is a single operator watching their own stack. ZKSplunk Macro is for ecosystem watchers who read only public-chain data to keep an eye on shared infrastructure (the cross-operator aggregation view is the part of Macro we are still building toward).

When a critical incident is confirmed, it can be anchored on-chain — and this is where the privacy-native design really shows. The on-chain record is anonymous and unlinkable: only the anonymized incident class, its severity, the epoch, a payload commitment, and a one-time nullifier go on-chain. The operator and the node are never published, so the whole network gains awareness of what went wrong without anyone having to expose who they are. An auditor can later re-hash the off-chain Splunk detail and prove it matches exactly what was attested. The contract is demo-wired and not yet audited, so we treat it as a working proof of the pattern rather than a production attestation system.

How we built it

We built ZKSplunk as working, type-checked code rather than a pitch deck, and we organized it around the DIDzMonolith convention of demoLand and zkMonitor. These are conventions designed by our team for quickly iterating through UI and the offline pipeline first, then attaching the live dependencies once the mock side is correct. The same core of connector, vitals, and contract code runs in both, and only the source of the data and the destination it flows to ever change.

The heart of the system lives in the connector. It is a production-grade Splunk HEC client with batching, exponential-backoff retry, and a heartbeat, wrapped by a forwarder that manages the full lifecycle of the connection. Around it sit a type-safe vitals adapter, a canonical telemetry-commitment hasher, and an attestation client. To teach Splunk to actually speak the language of zero-knowledge infrastructure, the connector ships with fourteen purpose-built field extractions and eleven SPL saved searches, plus the dashboards, the operator map, and the alert rules in the installable Splunk app.

The on-chain anchor is a Compact smart contract, zksplunk.compact, written for Compact v0.23 (pragma language_version >= 0.23). It uses a sealed ledger and public keys derived through persistent hashing, and it is built so that operators are recorded only as anonymous commitments in a HistoricMerkleTree and prove their membership in zero knowledge. It exposes five circuits: selfRegisterAsOperator and registerOperator for managing the anonymous operator set, attestCriticalIncident for anchoring a critical incident (it proves Merkle membership in ZK, spends a one-time scoped nullifier so the report is unlinkable and replay-proof, and appends an anonymized record), and the read-only getAttestationCount and isNullifierSpent views over the public log. The telemetry itself never goes on-chain — only the commitments and the anonymized incident class do.

The telemetry source is the full MidnightVitals module, built in React and TypeScript, with both a mock provider and a live HTTP-check provider hiding behind a single interface. A Midnight Indexer provider supplies public-chain data for the Macro lens, so ecosystem watchers can judge infrastructure health from public data alone.

The AI analyst is real and ships in two forms. Inside the Splunk app, the ZKSplunk AI Toolkit Analyst tab aggregates index=zksplunk evidence with SPL and calls the Splunk AI Toolkit directly via | ai prompt="{prompt}" provider=Gemini model=gemini-2.5-flash, so an operator can ask plain-language questions and get evidence-backed answers without leaving Splunk. Alongside it, a local ai-agent chat queries Splunk through the Splunk MCP server when configured (falling back to Splunk REST otherwise) and prefers the Splunk AI Toolkit for phrasing, with external LLMs as a fallback only. The current loop is: detect in Splunk, investigate via Splunk MCP/REST evidence, respond with operator guidance.

demoLand runs the zkZap attack scenarios completely offline and emits a self-contained dashboard with inline SVG and zero external dependencies, which makes it safe to show in a recorded demo. zkMonitor wires live HTTP vitals into a real Splunk Cloud HEC endpoint, with optional on-chain attestation through midnight.js and the Lace wallet. Both sides pass strict TypeScript compilation.

The piece we are most excited about — and the one we have laid the groundwork for — is a bridge between two Model Context Protocol servers, Splunk's and Midnight's, joined into a single bidirectional AI layer and grounded in the midnight-manual corpus. Today the implemented loop is the Splunk side: MCP/REST evidence plus the AI Toolkit. The designed next step adds the Midnight MCP server so an AI agent could notice a latency spike in Splunk and then cross over to the Midnight side to investigate why, correlating a proof-server slowdown with, say, an inefficient loop in a contract that is overwhelming the prover. Neither server can make that leap on its own; the bridge is the innovation, and it is the headline item on our roadmap.

Finally, we made it trivial to adopt. Integrating ZKSplunk into any Midnight DApp takes three lines: construct a forwarder, connect it, and wire its callbacks into the vitals provider. From that moment on, every vital check, log entry, and diagnostic report flows to Splunk automatically.

Challenges we ran into

The deepest challenge was philosophical before it was technical: you cannot watch what you cannot see. Midnight is private by design, and shielded state, circuit witnesses, and the parties and amounts of shielded transfers are invisible, and must stay that way. The hard design work was being rigorously honest about that line: build everything from the public surface and never claim to detect what is genuinely undetectable. Overclaiming would have been dishonest, and it would have betrayed the very privacy guarantee that makes the chain worth protecting.

Then there was the silence of failure. Unlike traditional chains that confirm in milliseconds, Midnight proofs take the better part of half a minute and can die mid-generation with no external signal at all; the proof servers are Docker containers running Halo 2 and UltraPlonk circuits that run out of memory and crash quietly. We had to invent telemetry semantics for failure modes that simply do not exist in any of Splunk's existing connectors.

We also had no prior art to lean on. Splunk has connectors for Ethereum, Hyperledger, and Quorum, and nothing whatsoever for zero-knowledge infrastructure. Every field extraction and saved search had to be designed from first principles by mapping ZK threat models onto the observable public evidence.

Anchoring incidents on-chain brought its own puzzle: how to prove integrity and broadcast an anomaly to the whole network without leaking who reported it or what the private data was. We solved it with an anonymous, unlinkable design — Merkle-membership proofs over operator commitments, a one-time scoped nullifier per report, and a canonical commitment scheme — so an auditor can independently re-hash the off-chain Splunk data and verify it against the on-chain commitment, all without ever putting sensitive telemetry, or operator identity, on a public ledger.

And underneath all of it was the simple pressure of two people building an entirely new observability category against a hard deadline. That forced ruthless prioritization: ship a working end-to-end pipeline first, keep everything strictly type-checked, make the demo reproducible offline so nothing could fall apart on stage, and be disciplined about separating what is built from what is designed and planned.

Accomplishments that we're proud of

We believe ZKSplunk is the first Splunk connector for any zero-knowledge blockchain on Earth, and that zkZap is the first privacy-native security-operations pattern built for one. That alone is a genuine first.

Just as important to us, it is real. The connector, the contract, the vitals module, the installable Splunk app, and both the demoLand and zkMonitor runners are all working and pass strict TypeScript compilation. There are fourteen field extractions, eleven SPL saved searches, ten alert rules, and a five-circuit Compact contract validated against the Midnight toolchain — not slideware.

We are proud that the AI analyst is genuinely shipped, not promised: the ZKSplunk AI Toolkit Analyst tab runs SPL over index=zksplunk and calls the Splunk AI Toolkit (| ai, Gemini gemini-2.5-flash) right inside Splunk, and the local ai-agent chat backs the same evidence-first loop through Splunk MCP/REST. The dual Splunk-plus-Midnight MCP bridge is the architecture we built toward and is the top of our roadmap; we kept it clearly labelled as the next step rather than dressing it up as done.

The on-chain attestation is the piece we keep coming back to, because anonymous, unlinkable incident broadcasting is exactly the privacy-native answer the inspiration demanded: the network learns what went wrong without anyone exposing who they are. We made the demo honest too: the offline dashboard has no CDNs and no external dependencies, so it is fully reproducible and tamper-free.

Most of all, we closed the gap that inspired us. We took a real, painful lesson from the November 2025 Cardano chain split — detection-to-coordination measured in hours rather than seconds — and built the early-warning and operator-notification layer we wished had existed, on a platform enterprises already trust. And because any Midnight DApp can integrate in three lines, ZKSplunk is not a silo but a spine: the operational backbone for the entire DIDzMonolith ecosystem.

What we learned

We learned, first, that privacy and observability are not enemies. The instinct is to assume a privacy chain cannot be monitored, and we found the opposite to be true: the public surface (metadata, volumes, transaction status, block cadence, DUST health) is a rich and honest signal you can genuinely defend a network with, as long as you respect the line and never reach for private state.

We learned that decentralization needs a coordination channel. The Cardano split made it painfully clear that a fix being ready and a fix reaching every operator are two very different problems, and that the detection-to-coordination loop is the weak link. A pub-sub notification layer, which Splunk's alert actions already provide, closes most of that gap today.

We learned that ZK observability is its own discipline. Proof lifecycles, silent prover failures, and shielded-state semantics have no equivalent in the world of HTTP-200 monitoring, and translating that domain into Splunk's language showed us just how much purpose-built tooling this ecosystem still needs.

We learned that honesty is a feature. Drawing a hard, explicit boundary around what is and is not detectable — and being equally explicit about what is built versus what is planned — made the project stronger, not weaker. It is exactly what makes the security claims credible to an auditor and the roadmap credible to a judge.

We learned that AI is most useful when it is grounded. Calling the Splunk AI Toolkit over real index=zksplunk evidence, rather than free-associating, turned the analyst into something that actually reasons over telemetry; and designing the bridge to a Midnight MCP server grounded in the midnight-manual corpus showed us where the next leap in value lives — correlation across telemetry and contract source at once.

And we learned that the demoLand and zkMonitor split is a quiet superpower. Sharing one core between an offline-safe simulation and a live deployment let us build, test, and demonstrate with confidence, without standing up full infrastructure every single time.

Sources

Intersect, Incident Report: Network Partition Analysis and Resolution Strategy (post-mortem)
CoinDesk, "Cardano Temporarily Splits Into Two Chains..." (Nov 23, 2025)
Bitfinex Blog, "What Actually Happened with the Cardano Exploit?"
Charles Hoskinson, public statements on X and livestream commentary following the November 2025 chain split
CryptoRank, "Cardano Issues First Report on Mainnet Partition as Hoskinson Calls for Unity"

Dates and figures reflect public reporting at the time of writing. The technical lesson, not the attribution of intent, is what motivates this project.

ZKSplunk- "Splunking OPSEC with Midnight Network"