neo-finds-evil: The Analyst Who Never Forgets
What it does
A senior incident responder does not solve a case by staring at one artifact. They remember.
They remember that the suspicious account seen on the DMZ host also appeared in a failed logon burst three days earlier. They remember that a PowerShell command on one RDP server looks like the same cradle used on a workstation. They remember that a credential-theft attempt matters more when it appears after lateral movement, not before. That memory is what turns tool output into an investigation.
Most AI-driven DFIR systems still lack that memory. They can run tools quickly, but every command produces another blob of text for the model to read, compress, correlate, and eventually forget. As cases scale from one machine to five, ten, or twenty, this becomes expensive, slow, and risky. Context fills up. Correlation moves into the model's short-term memory. Hallucination becomes more likely exactly when the investigation becomes more important.
neo-finds-evil was built around a different idea: do not make the model memorize the case. Give the case a structure the model can query.
The project extends Protocol SIFT with a read-only forensic graph layer. Protocol SIFT already lets a Claude Code agent use the SANS SIFT Workstation to run real DFIR tools against real evidence. neo-finds-evil adds a Neo4j graph that represents the whole investigation across hosts, artifacts, and time. Processes, logons, files, registry changes, network connections, PowerShell script blocks, services, accounts, domains, and source events become connected evidence, not isolated text dumps.
The result is an agent that can ask senior-analyst questions:
- Where did this account appear across all hosts?
- Which machines reused the same suspicious PowerShell payload?
- What process chain led to this connection?
- Can this finding be traced back to a specific source event?
- Is this confirmed, inferred, or unsupported?
For the hackathon dataset, we used the SANS FOR508 SRL-2018 / SHIELDBASE case: seven Windows hosts with disk and memory evidence, converted into a graph of roughly 359,788 nodes and 1.9 million relationships. On top of that graph, the agent reconstructed a hands-on-keyboard intrusion involving brute-force activity, PowerShell Empire-style C2, WMI persistence, lateral movement through PsExec and services, credential tooling, and an attempted domain-controller SAM-hive copy.
neo-finds-evil is a proof of concept for a practical defensive pattern: let SIFT tools extract truth from artifacts, let a graph preserve the relationships, and let the AI reason over a bounded, auditable, read-only view of the case.
The goal is not to replace the analyst. It is to give the analyst an autonomous partner that remembers the whole incident, admits uncertainty, corrects its mistakes, and can explain exactly why it believes what it believes.
How we built it
neo-finds-evil adds one main component to a Protocol SIFT deployment: a project-scoped MCP server named forensics-graph. Architecturally this is a Custom MCP Server pattern: the novel capability is delivered as a read-only MCP server exposed to the same agent, and the security boundary is the tool surface itself rather than a prompt.
At investigation time, the Claude Code agent runs under Protocol SIFT's DFIR operating posture and can still use SIFT's per-artifact tool layer for Volatility, Plaso, Sleuth Kit, EZ Tools, YARA, and other SIFT workflows. In parallel, the same agent receives five read-only graph tools exposed by forensics-graph:
list_hunts: lists vetted correlation hunts.run_hunt: executes a named, bounded, vetted hunt.get_host_summary: returns anchored per-host counts and relationship summaries.get_event: resolves a finding to a specific sourceWindowsEvent.query_graph: provides a guarded read-only Cypher escape hatch for ad-hoc reasoning.
The Neo4j graph is built out-of-band by the ingest pipeline. Evidence extraction and graph writes are not available to the agent. The agent only reads the completed graph.
The high-level data flow is linear: disk and memory evidence feeds an out-of-band ingest pipeline that builds the Neo4j forensic graph; at investigation time the agent reaches that graph only through the read-only forensics-graph MCP server, while still using Protocol SIFT's per-artifact tools directly; and the agent's output is a traceable investigative narrative. (A rendered architecture diagram is in the repository at docs/architecture.png.)
The graph contains hosts, Windows events, processes, files, registry keys, services, user accounts, domains, IP addresses, and script blocks. Relationships include REPORTED, RAN_ON, SPAWNED, CONNECTED_TO, LOGGED_IN, MODIFIED_REGISTRY, CREATED_FILE, RESOLVED_DNS, INSTALLED_SERVICE, and related case-specific links.
This gives the agent a compact, structured memory of the case. Instead of loading millions of raw lines into the model context, the agent can retrieve the specific slice of evidence needed for the next reasoning step.
The agent also cannot damage the case. Graph access is read-only by construction: there is no MCP write tool. The free-form query path runs in a Neo4j read transaction, rejects write clauses, enforces limits and timeouts, and logs every call. Any finding can be followed back through the audit trail to the exact tool execution and then to the original event source.
The design rests on seven decisions and their tradeoffs.
1. Graph correlation instead of model-context correlation
The main design decision was to move correlation out of the LLM context window and into a graph database.
Why it matters:
- It keeps token usage bounded even when the case contains hundreds of thousands of artifacts.
- It lets the agent ask relationship questions directly, such as account-to-host, process-to-network, or script-to-host correlations.
- It reduces hallucination pressure because the model receives targeted evidence instead of noisy bulk output.
- It scales better to realistic incident response cases with many hosts and mixed disk, memory, event-log, and network artifacts.
This directly supports the hackathon goal of making autonomous response practical against machine-speed adversaries.
2. Protocol SIFT remains the per-artifact expert layer
We did not try to replace Protocol SIFT. Protocol SIFT is strong at driving SANS SIFT tools against individual artifacts. neo-finds-evil adds the missing cross-host and cross-time layer.
Why it matters:
- The submission builds on the required SIFT ecosystem instead of bypassing it.
- Existing DFIR tools remain the source of artifact-level truth.
- The graph layer becomes a second analytical surface over the same evidence.
- The project is a genuine extension of Protocol SIFT, not a disconnected agent demo.
In short: SIFT gives the agent forensic hands. neo-finds-evil gives it investigative memory.
3. Read-only MCP tool surface
The MCP server exposes exactly five read-only tools. There is no ingest tool, no mutation tool, no graph-update tool, and no administrative Neo4j tool.
Why it matters:
- The agent cannot alter evidence or graph state through MCP because no write action is expressible.
- This is an architectural guardrail, not a prompt instruction.
- It aligns with forensic integrity requirements and the hackathon's "Constraint Implementation" criterion.
- It lets us safely provide powerful graph access without trusting the model to avoid dangerous operations.
The strongest security boundary is the tool surface itself: write capability does not exist.
4. Read transactions and write rejection for free-form queries
The query_graph tool is intentionally flexible because real investigations need ad-hoc questions. But flexibility is bounded.
query_graph runs inside a Neo4j read transaction. It also applies a lexical write/DDL pre-check, mandatory result limits, server-side timeouts, and Python-side row caps.
Why it matters:
- The agent can ask new investigative questions without waiting for a developer to add a named hunt.
- Write attempts such as
CREATE,MERGE,SET,DELETE, orDETACH DELETEare rejected. - Long-running or unbounded queries cannot consume the environment indefinitely.
- False rejection of a suspicious-looking read query is acceptable; allowing a write is not.
This gives the project both autonomy and safety.
5. Event-level traceability with get_event
Every event-level finding can be resolved back to a source event ID, and get_event returns the underlying WindowsEvent including source channel information such as Security.evtx.
Why it matters:
- Findings are not just model statements; they are anchored to source artifacts.
- A report claim can be traced back to a tool call and then to the original evidence.
- The agent can distinguish confirmed facts from inferences.
- Self-correction becomes possible because claims can be re-checked against raw or source-level artifacts.
This is critical for IR accuracy and audit trail quality.
6. Typed errors as a self-correction interface
Tool failures are returned as structured errors such as unknown_hunt, unknown_host, not_found, timeout, write_rejected, or query_error, with remediation guidance.
Why it matters:
- The agent can recover from bad assumptions without human intervention.
- Empty results can be treated as possible schema, coverage, or hypothesis problems rather than final answers.
- The execution log records the correction path.
- The demo can show real self-correction: empty query, schema discovery, corrected query, traceable source event.
This turns failure handling into part of the autonomous workflow rather than an operator task.
7. Out-of-band ingest and agent-time read-only investigation
The graph is built before investigation time. Phase-1 ingest reads evidence and creates graph relationships, but that pipeline is not exposed to the agent.
Why it matters:
- The write path is outside the autonomous agent.
- A known graph state can be restored to reproduce the investigation.
- The agent's behavior is easier to audit because every investigation-time graph action is read-only.
- The architecture separates evidence processing from investigative reasoning.
This is a practical compromise: powerful preprocessing, safe autonomous investigation.
Challenges we ran into
But the real story is reliability.
The agent did not simply produce a polished incident report. It corrected itself. In one run, an initial query for spsql activity returned zero rows because the agent guessed the wrong property names. Instead of concluding there was no activity, it discovered the graph schema, sampled relevant nodes, rewrote the query, found cross-host activity, and pulled a specific failed-logon event back to Security.evtx. That sequence is preserved in the MCP audit log.
In another finding, the agent initially over-attributed WMI persistence to SRL-FILE. It then challenged its own claim, checked raw artifacts and positive-control hosts, and retracted that host from the persistence finding. In the SAM-hive case, it confirmed that the PowerShell command ran, but refused to claim the copy succeeded because the graph did not contain enough corroborating telemetry. That distinction matters: a useful DFIR agent must know the difference between evidence, inference, and speculation.
That read-only claim was itself put on trial. Late in the project we ran an independent adversarial audit of the codebase, then built a live regression test that bypasses the lexical guard and fires real write statements through the same path the agent's queries use. The Neo4j server rejected both, exactly as designed. But the test's sanity check (verifying it would fail if the guard were removed) disproved something our own documentation claimed: the docstring said the read transaction was 'never committed,' and it turned out the driver auto-commits on clean exit. The access-mode enforcement was the only real backstop all along. We corrected the documentation, and the regression test now permanently guards the one layer that actually holds. The system caught its own false belief about its own safety, and the correction is preserved in the git history. We think that is the most honest demonstration of the project's values in the whole repository.
We did not only grade ourselves. The agent's adjudication behavior was benchmarked against VIGIA, an independent intent-adjudication evaluation framework by Anna Tchijova, using her published vigia-cases dataset. On the scoreable cases the agent achieved 100% verdict accuracy with a 0% false-positive rate and a 0% missed-malice rate, and it passed the specificity gate: presented with a case designed to tempt over-attribution, it said suspicion, not malice. Those numbers come with two honest caveats, both disclosed in the repository rather than discovered by a reader. First, the sample is small (three verdict cases and one false-positive-eligible case), so the headline reads as consistency, not statistical precision. Second, the agent failed VIGIA's TTP-coverage bar: it labelled only 14.3% of expected techniques exactly (42.9% at the technique-family level) against a 60% target. The verdicts were right; the MITRE labelling underneath them is the documented weak spot. We report it because a benchmark you only publish when it flatters you is not a benchmark.
One more disclosure in the same spirit: the SRL-2018 case is the well-known SANS FOR508 teaching dataset, and write-ups of it exist online. We cannot rule out that the underlying model has seen discussions of this intrusion in training. The benchmark design mitigates this (VIGIA cases are scored on adjudication behavior, not on recalling the SHIELDBASE answer key), but the caveat belongs in the open.
What we learned
A few lessons shaped the final design.
Reliability is a property you engineer, not one you assume. The behavior that mattered most was the agent treating an empty result as a question rather than an answer, and abstaining when the telemetry was not there to support a claim. The tooling has to make abstaining the easy path: typed errors that suggest what to do next, hunts that return bounded results, and a get_event call that forces a finding back to a source artifact.
The read-only guarantee has to live in the architecture, not in a prompt, and you cannot take your own safety claims on faith. Our adversarial audit disproved a claim our own documentation made about how the read transaction behaved; the access-mode enforcement was the real backstop all along. The lesson we encoded: regression-test the boundary that actually holds, and write down what you could not prove.
Honest reporting beats a flattering number. We published the VIGIA benchmark including the bar we failed, because a benchmark you only publish when it flatters you is not a benchmark, and because for a defensive tool the cost of an overconfident wrong answer is higher than the cost of an admitted gap.
A small operational note for reproducers: during testing we found that Anthropic's Claude Fable 5 routes DFIR and forensics workloads to Claude Opus 4.8 (a visible model switch, not a silent one), so this project effectively runs on Opus-class models end-to-end.
What's next
A few directions follow naturally from the limitations we documented.
Tighten MITRE ATT&CK labelling to sub-technique precision. The VIGIA benchmark showed the verdicts are right but the technique labelling underneath them is the weak spot; closing the parent-versus-sub-technique gap is the most direct accuracy win on the board.
Extend per-event traceability to every hunt. Per-event hunts already carry an event id and source channel for direct get_event traceback; the aggregate and count hunts return a host plus key sufficient for a follow-up but are not yet per-event by nature. Generalizing the event-id projection across all of them would make every reported row individually resolvable to a source artifact.
Bring Protocol SIFT's own tool layer under the same architectural guardrail. SIFT's forensic CLIs are governed today by a Bash allow/deny list, which is config-enforced rather than tool-architectural. Wrapping that layer as MCP tools would extend the same "no dangerous action is expressible" property from the graph to the whole stack.
Harden the boundary for portability. The hunts are expressed in Cypher and the tool contract is engine-agnostic, so porting to another graph engine is mostly a matter of re-expressing the queries; adopting a true read-only database role, outside Neo4j Community Edition's single-user constraint, would add a second enforced layer beneath the tool surface.
Evaluation criteria coverage
Autonomous Execution Quality
Covered by:
- Schema-discovery and query-rewrite behavior when the first
spsqlquery returned zero rows. - Typed MCP errors that tell the agent what went wrong and how to recover.
- Empty-hunt handling where the agent treated zero rows as a possible coverage issue, not an all-clear.
- Self-correction of a finding-level mistake: retracting
SRL-FILEas a WMI persistence host after raw-vs-graph verification.
Why it matters: the agent does not just run commands. It reasons about failed assumptions, changes approach, and preserves the correction trail.
IR Accuracy
Covered by:
- Explicit confirmed-vs-inferred-vs-abstained language in findings.
get_event(event_id)traceability to sourceWindowsEventrecords and source channels.- Honest reporting of known gaps, such as unparsed Sysmon EID 10 process-access fields.
- Abstaining from claiming SAM-hive copy success when telemetry only confirmed command execution.
- Independent benchmark against Anna Tchijova's VIGIA framework: 100% verdict accuracy, 0% FPR, 0% missed malice on the scoreable cases, with the small-n basis and the failed TTP-coverage bar (14.3% exact) disclosed in
docs/accuracy-report.mdrather than omitted.
Why it matters: the project rewards defensible findings over confident speculation.
Breadth And Depth Of Analysis
Covered by:
- Seven-host Windows enterprise case.
- Disk and memory evidence.
- EVTX, Volatility artifacts, shimcache, prefetch, MFT/path catalog, registry, bulk_extractor output, PowerShell script blocks, process, network, file, service, and account relationships.
- Cross-host findings involving brute force, C2, persistence, credential theft, lateral movement, and baseline/noise separation.
Why it matters: the submission demonstrates depth on a realistic enterprise intrusion rather than shallow coverage of isolated artifacts.
Constraint Implementation
Covered by:
- MCP server with no write tools.
- Neo4j read transactions for free-form graph queries.
- Write/DDL lexical rejection as defense in depth.
- Mandatory limits, timeouts, and row caps.
- Out-of-band ingest so the agent has no graph construction or mutation path.
- A live regression test fires real writes through the read path against the running graph and asserts the genuine server-side AccessMode rejection: the guarantee is regression-tested, not asserted.
Why it matters: the read-only boundary is enforced by architecture, not just by telling the model to behave.
Audit Trail Quality
Covered by:
- JSON-lines MCP audit log containing timestamp, tool name, arguments, duration, status, and row count.
- Tool statuses such as
ok,write_rejected,timeout,unknown_host, andnot_found. - Finding traceability from audit line to hunt row to
event_idtoget_eventto source event channel. - Published execution-log sample showing a real self-correction sequence, plus a sanitized per-turn token-usage log from the recorded session.
Why it matters: the reasoning path is visible, and every conclusion can be verified from the execution record.
Usability And Documentation
Covered by:
- README with setup and a step-by-step local runbook.
- Project-scoped
.mcp.jsonregistration. - Restorable Neo4j graph release asset (v1.0, with a published SHA-256).
- Documented architecture diagram.
- Evidence dataset documentation.
- Accuracy report.
- Execution logs.
- Apache-2.0 license.
Why it matters: another practitioner can run, inspect, reproduce, and extend the system.
Technical summary
neo-finds-evil is a read-only Neo4j graph-correlation layer for Protocol SIFT. It converts a multi-host forensic case into a structured graph and exposes that graph to a Claude Code agent through five read-only MCP tools. The design keeps raw artifact volume out of the model context, lets the agent reason across hosts and time, and preserves a traceable chain from each finding back to source evidence.
The project's core contribution is not another parser. It is an autonomous investigation pattern:
- SIFT tools provide artifact-level extraction and expertise.
- The graph preserves cross-artifact and cross-host relationships.
- The MCP server enforces read-only, bounded, auditable access.
- The agent performs reasoning, self-correction, and narrative synthesis.
This architecture helps close the speed gap the hackathon is focused on while preserving the qualities that matter in DFIR: evidence integrity, traceability, accuracy, and honest uncertainty.
Log in or sign up for Devpost to join the conversation.