AgentSphere SecureOps

AgentSphere is an agentic cybersecurity workflow that uses Web3 to make AI decisions accountable, verifiable and resistant to tampering.

Comment

Inspiration

A SOC analyst's real problem is not a lack of opinions—it is that they cannot afford to trust an unsupported one. Microsoft's GUIDE dataset makes the scale of that problem concrete: our working set contains 5,000 incidents and 591,340 evidence rows, and a single incident can contain more than a thousand rows of telemetry. Alert fatigue is not merely a user-experience problem; it is an arithmetic one.

The obvious response in 2026 is to point an LLM at the queue. We believe that only moves the problem. An agent that says, “This is a true positive,” has replaced an unread alert with an unaccountable opinion. Three questions determine whether an autonomous SOC workflow is usable at all:

  1. Can it show its work? Not just a confidence score, but the evidence it cited, the prompt it received, and what it was deliberately prevented from seeing.
  2. Can it be told no? Autonomy must stop at a boundary that a prompt cannot negotiate its way around.
  3. Can we prove that the record was not edited afterward—even by the people operating the system?

AgentSphere SecureOps is our attempt to answer all three questions—and to be honest about the places where our first answers fell short.

What it does

AgentSphere sends a Microsoft GUIDE security incident through six bounded AI agents, applies a deterministic policy gate that can overrule them, and anchors the resulting decision as tamper-evident proof on Ethereum Sepolia.

Detection → Correlation → Investigation → Triage → Remediation → Verifier → Policy Gate

  • Union-Find correlation reduces alert overload before any model reasons about the case. In our flagship incident, 26 alerts become 18 clusters—a 31% reduction. The result is deterministic, inspectable, and impossible for an agent to invent.
  • The Verifier can reject the work of the other agents. It produces none of the outputs it evaluates, and seven structural checks run on every backend, including live model output.
  • The policy gate is code, not a prompt. POL-001 limits action risk, POL-004 enforces a confidence floor, POL-005 respects the Verifier's verdict, and POL-006 blocks finalisation if any stage falls back to a conservative default. An agent cannot talk its way past these rules.
  • Every completed decision can be anchored as two 32-byte digests. The Proof screen exposes the complete journey: a SQLite record → canonical JSON → keccak256 → a signed transaction → a Sepolia block → a struct in contract storage.

All remediation is simulated. AgentSphere does not isolate devices, disable accounts, revoke real sessions, or otherwise act on production systems. That limitation is stated throughout the application.

How we built it

Data pipeline

GUIDE is evidence-level data: evidence rows roll up into alerts, and alerts roll up into incidents. A single schema.py module acts as the column contract, allowing synthetic fixtures and the real 2.4 GB dataset to move through the same pipeline.

We also mask sentinel values before downstream processing. One device identifier accounted for 97.85% of its column; a model that learns from that value has learned a data artifact, not security behavior.

Bounded agent workflow

We froze the agent contracts on day two and test them against the schemas stored in artifacts/schemas/. Each agent has a narrow role and must return validated structured output.

No ground-truth label reaches a pre-decision agent. Retrieval can return similar incidents, but never their analyst labels. Allowing either would leak the answer into the prompt and invalidate every metric we report.

Web3 accountability layer

We built two Solidity contracts:

  • AgentRegistry determines which agent identities are authorized to submit decisions.
  • DecisionProof stores decision digests and approval state—and nothing else.

Each agent role has its own registered identity, so an on-chain decision records which authorized role submitted it. Sensitive evidence, prompts, and reasoning remain private and off-chain.

Only hashes can cross this boundary. This is enforced by the contract's interface, not by an application policy that could later be relaxed: DecisionProof has no function or storage capable of accepting an evidence row.

bytes32 fingerprint = keccak256(
    abi.encodePacked(incidentId, evidenceHash, outputHash)
);
uint256 existing = _fingerprints[fingerprint];
if (existing != 0) revert DuplicateDecision(existing);

Graph analysis

GUIDE does not ship with a graph, so we construct one from evidence co-occurrence within each alert. We deliberately avoid connecting every entity within an incident; doing that for an incident with 1,313 evidence rows would create a misleading, nearly complete graph.

We use Dijkstra's algorithm to find the most probable attack path with edge cost

$$ w(e) = -\log\big(\operatorname{confidence}(e)\big) $$

rather than (1-c). A linear cost can prefer a long path containing one extremely weak link. Under the logarithmic cost, minimizing the total path weight is equivalent to maximizing the path probability (\prod_i c_i), which better represents the “most probable” path.

Traversal is depth-capped and never expands through a hub. The largest hub in our real graph is process:6, with degree 1,025; an uncapped breadth-first search from that node reaches most of the graph.

Reproducible demonstrations

Our hermetic replay cache is content-addressed using the model, prompt version, system prompt, user prompt, and response schema. A replay cache miss raises an explicit error instead of silently becoming a paid live call. This keeps the recorded demonstration fast, reproducible, and faithful to the output we tested.

Challenges we ran into

The blockchain refuses to forget

submitDecision prevents duplicate submissions, so re-anchoring an identical incident resolves to its existing decision fingerprint permanently. Initially, the Proof screen displayed dashes for its block and gas information, making a valid existing record look as though nothing had been written on-chain.

We fixed this by finding the original transaction through its digest and describing it honestly, instead of implying that a new transaction had just been submitted.

Our own prompts were the bug

The Verifier escalated 97.5% of live incidents, compared with 5.0% in deterministic mode, even when every structural check passed. It appeared cautious, but the actual cause was context starvation: its prompt rendered len(cited) instead of the cited evidence itself. We were asking it to certify evidence it could not see, so it correctly refused.

This was a valuable lesson: an agent deprived of context often does not crash. It produces plausible caution, which is much harder to diagnose.

We manufactured the evidence gaps we later escalated

Our correlation stage originally emitted one “missing information” entry for every absent entity type, checking all seven entity types against every incident. Across the corpus, this produced an average of 5.26 artificial gaps per incident. Only 5 of 5,000 incidents contained all seven entity types, while 53.2% contained one type or none.

Those gaps flowed through four downstream prompts, and the Verifier treated more than three gaps as a reason to challenge high confidence. In other words, normal incidents reached the gate already described as incomplete. After correcting the logic, the mean fell from 5.26 to 0.53 genuine gaps per incident.

Some charts were visualizing constants

Our initial provenance charts looked broken because the underlying values barely varied. Across 25 sampled incidents, confidence spread within an attack path was 0.000 every time, and 19 of 25 blast radii reached exactly one entity type.

No styling can rescue a chart of a constant. We replaced those charts with the actual path and labeled evidence rows, making the result clearer while reducing the page bundle from approximately 410 kB to 36 kB by removing the charting library.

Reporting results we did not like

On the OpenAI backend, our evaluated agent chain achieved 0.4084 macro F1, compared with the baseline's 0.4669. We show that result in the interface.

The agent layer currently adds evidence-grounded explanation, independent verification, policy enforcement, and accountability—not higher classification accuracy. Hiding that distinction would undermine the trust problem we set out to solve.

Accomplishments that we're proud of

Tampering that is visible from the back of the room

With one button, the application performs the action an insider with database access might take: a single UPDATE to agent_runs.output_json, without changing a stored hash or sending a transaction.

The integrity verdict immediately changes to TAMPERED, and the digest grid shows 32 of 32 bytes changed. This makes the avalanche property of the hash visible in a way that a shortened value such as 0x8f3a…c41d never could.

The verification earns that verdict. AgentSphere recomputes both digests from the underlying stored evidence and agent outputs every time; it never trusts a stored hash column. An earlier implementation compared a stored hash with the chain, which meant editing an agent output could still verify successfully because the comparison never returned to the source data. We found and corrected that flaw.

A real end-to-end anchor

We anchored and finalized a real decision on Sepolia: decision #29, in block 11,481,573, using 201,393 gas, with a public explorer link available directly from the Proof screen.

A non-LLM baseline that beats our agents—and remains visible

Our LightGBM baseline achieved 0.7072 accuracy, 0.6774 macro F1, and 0.6092 true-positive recall on held-out incidents. Reporting the stronger baseline beside the agent output is what makes the rest of our evaluation credible.

Honesty as a design constraint

The WitFoo dataset contains 9,552 incidents, and every one is labeled malicious. We removed the threat-label filter and explain why, instead of shipping a control that could only return all records or none. When a value is unknown, the interface displays a dash. We do not use placeholder transaction hashes, blocks, or metrics.

What we learned

A verifier that rejects everything is as unhelpful as one that accepts everything. Left uncalibrated, the live Verifier rejected 40 of 40 real incidents because of perceived evidence gaps. Calibration is not a finishing detail; it determines whether verification is useful at all.

Silent context starvation is a dominant failure mode in agent chains. Our two most serious agent-workflow bugs had the same cause: an agent was asked to reason about information it had not received. Neither failure raised an exception. Both looked like reasonable caution.

Reproducibility and determinism are different claims. Replay mode and our deterministic fallback are reproducible and tested as such. Live reasoning is not bit-reproducible—the routed models expose no temperature, top_p, or seed controls—so we measure and publish variance rather than claiming determinism we do not have.

A blockchain is not a database with extra steps—but only if its unique guarantees are explicit. For AgentSphere, those guarantees are rejecting unauthorized writers at the storage layer, preventing finalization of risky decisions without human approval, and detecting tampering by the party operating the application database. If a control does not require independent infrastructure, it does not belong on-chain.

What's next for AgentSphere SecureOps

  • Resolve the remaining escalation honestly. The manufactured evidence gaps are gone and POL-004 now passes on our flagship case, but the Verifier still escalates on substantive questions of proportionality and confidence calibration. That is a defensible outcome, not a demo failure, and we would rather report it than tune thresholds until the result looks flattering.
  • Recover complete anchor history from contract events. The DecisionSubmitted event already contains the transaction and block information; we want recovery to work even when no local transaction history survives.
  • Expand the offline replay set. The hermetic cache currently covers the six curated demo incidents. Pre-validating the full 30-case showcase pool would make any showcase incident reproducible offline.
  • Run a second dataset through the complete agent workflow. The graph layer already runs unchanged on WitFoo's shipped provenance graph of 35,133 nodes and 634,190 edges. The next step is to bring the agent layer across the same boundary.
  • Make approval a first-class on-chain flow. We plan to hash and anchor the analyst's approval comment so the claim that “a human approved this” is cryptographically verifiable rather than merely asserted.

AgentSphere SecureOps is not an attempt to replace the analyst. It is an attempt to give analysts an AI workforce whose reasoning is visible, whose authority is bounded, and whose history cannot be silently rewritten.

Built With

Share this project:

Updates

Submission history