Grounded Autonomous DFIR Agent

Thesis

Find Evil is an autonomous incident-response agent whose spine is grounded finding-confirmation. Every finding the agent drafts is adjudicated against the exact tool output that supports it — classified grounded, ungrounded, or contradicted — and a contradicted finding is re-investigated with a different tool before it is allowed into the report. The whole loop runs over a forensic MCP surface that is read-only by construction, so the guardrails are architectural, not prompt-based.

Find Evil's distinguishing choice is finding-level grounding: it adjudicates each individual finding against the evidence that produced it — not merely which tool to run — and backs the design with a measured accuracy number. A contradicted finding is re-investigated with a different tool rather than re-running the same one.

What it does

Find Evil drives a SANS SIFT Workstation through Protocol SIFT to investigate a disk, memory, log, or network artifact and produce a findings.json report where every finding carries its own provenance chain: finding → tool → command → output SHA256 → line span → timestamp.

The agent has two roles:

• Investigator — plans the investigation, calls forensic tools over MCP, and drafts findings, each one citing the specific tool execution that supports it.
• Skeptic — for each draft finding, re-fetches the cited tool output by hash and renders a verdict: grounded (enters the report), ungrounded (dropped as a hallucination), or contradicted (the output actively refutes the claim).

A contradicted finding does not trigger a re-run of the same tool. It dispatches a different tool or artifact to corroborate — for example, a registry-run-key persistence claim that a tool contradicts gets checked against Prefetch or Amcache — then reconciles the result and records the resolution in the finding's corroboration field. This is grounded self-correction, not blind retry.

Underneath sits a custom read-only forensic MCP server exposing 14 typed tool wrappers: volatility3 (malfind, psscan, pslist, pstree, dlllist, handles, cmdline, netscan), YARA, MFTECmd, EvtxECmd, RECmd, PECmd, and AmcacheParser. No generic shell tool exists. Each wrapper declares a pydantic input model, builds an argv list (never a shell string), runs the binary with shell=False, and JSON-parses the output before it reaches the model. Every call is hashed and written to an append-only audit log — the same hash a finding's provenance later resolves against.

How we built it

The agent loop is Claude Code extending the Protocol SIFT base, with the Skeptic as the novel layer. The pattern is a generate-then-adversarially-verify pipeline: a recall-biased discovery pass drafts findings, and a precision-biased skeptic adjudicates each one against its cited evidence and emits structured output.

The orchestrator runs run_tool → draft → finalize, where finalize is skeptic → corroborate → schema-validated emit. Components:

• MCP server (mcp_server.py) — stdio server exposing only the typed, gated wrappers.
• Tool wrappers (tools/) — each subclasses ReadOnlyTool; the base class handles safety-gating, subprocess, audit logging, and output capture.
• Safety layer (safety.py) — guard_argv (binary deny-list + shell-metacharacter rejection) and assert_read_only_path (no writes under /cases/, /mnt/, /media/, /evidence/). This is belt-and-suspenders with a Claude Code settings.json write-deny, so two independent layers both have to fail before evidence can be mutated.
• Audit log (audit.py) — append-only JSONL: timestamp, argv, output SHA256, token estimate, exit code, duration.
• Skeptic (skeptic.py + prompts/skeptic_system.md + model_client.py) — batched per-finding adjudication.
• Corroboration (corroboration.py) — different-tool selection by category-affinity plus hints, resilient to unavailable tools, then reconciliation.

The full loop runs offline against bundled evidence fixtures using the local claude CLI as the model backend (--allowed-tools "" so the nested call has no tools) — no SIFT box and no API key are needed to run the 32-test suite or the demo.

Accuracy results

The hackathon scores IR accuracy (correctness, hallucination detection, finding confirmation) and audit-trail quality. We measured the component that drives both — the finding-confirmation skeptic — against a no-skeptic baseline that reports every draft, on two complementary evals.

External (DFIR-Metric). DFIR-Metric (Cherif et al., ICONIP 2025 spotlight) is a third-party, peer-reviewed benchmark — neither its evidence nor its answers are ours. From its CTF module we took every challenge whose answer is a single source IP (n=9) and posed the skeptic two claims per challenge over the same logs: the true malicious IP (should ground) and a benign IP from the same logs (a hallucination trap, should be rejected).

Configuration	Precision	Recall	FP-rate	F1
Baseline — no skeptic	0.50	1.00	1.00	0.667
With finding-confirmation skeptic	1.00	0.78	0.00	0.875

All 9 benign trap IPs were rejected. The recall cost (2 of 9 true positives marked ungrounded) is the precision-first behavior the design and the self-correction literature predict; we report it rather than tune it away.

Controlled (incident_2026). A recall-biased investigator simulated by a fixed labeled draft set over a synthetic incident: 16 drafts = 8 true positives + 8 negatives (6 false-positive traps + 2 decoys), evidence replayed from fixtures (memory malfind, YARA, Windows event logs, registry).

Configuration	Precision	Recall	FP-rate	F1
Baseline — no skeptic	0.50	1.00	1.00	0.667
With finding-confirmation skeptic	1.00	1.00	0.00	1.00

All 16 drafts adjudicated correctly. Notably, the "svchost.exe PID 1408 is a clean service" trap was returned as contradicted (not merely ungrounded) — the correct routing for a claim a tool output actively refutes, which is what the corroboration step exists to recover.

The honest headline: across a third-party benchmark and a controlled case, the finding-confirmation layer drove the false-positive rate from 1.0 to 0.0 (precision 0.5 → 1.0), at a recall cost of zero on the controlled set and 2/9 on the external set.

How it maps to the six judging criteria

1. Autonomous Execution Quality (tiebreaker) — grounded self-correction is the spine: a contradicted finding is re-investigated with a different tool and reconciled before reporting. This follows the literature's clear finding that intrinsic self-correction degrades without an external oracle (CRITIC; "LLMs Cannot Self-Correct Reasoning Yet"); our oracle is the cited tool output.
2. IR Accuracy — measured precision 1.00 / FP-rate 0.00 on both evals, with hallucinated findings dropped (ungrounded) and refuted findings routed for corroboration (contradicted).
3. Breadth/Depth — 14 read-only wrappers spanning memory (volatility3 suite), disk/MFT (MFTECmd), logs (EvtxECmd), registry (RECmd, AmcacheParser), execution (PECmd), pattern-matching (YARA), and network (netscan). Depth on the confirmation loop over breadth on tool count.
4. Constraint Implementation (architectural, not prompt-based) — read-only is enforced at two independent layers: the MCP wrapper (binary deny-list, shell=False, argv-as-list, protected-path guard) and Claude Code settings.json write-deny. The agent physically cannot issue rm, dd, shred, wget, curl, or ssh.
5. Audit Trail Quality — every finding's provenance chain (tool → command → output SHA256 → line span → timestamp) is the audit trail; a finding cannot be emitted unless it cites a concrete tool execution, and the skeptic re-fetches that exact output by hash to adjudicate.
6. Usability/Docs — offline demo and 32-test suite run without a SIFT box or API key; README.md, docs/ARCHITECTURE.md, and evaluation/ACCURACY_REPORT.md document the design, security boundary, and reproducible measurement.

Challenges

The literature is blunt that LLMs are unreliable self-validators — intrinsic self-correction can make reasoning worse, not better. The design consequence shaped the whole build: self-correction here is never pure introspection; the skeptic re-fetches the exact tool output by hash and adjudicates against it, and corroboration reaches for a different tool rather than re-running the same one.

Grounding had to be mechanical, not vibes. A finding cannot exist without citing a concrete tool execution, and the cited output is content-addressed by SHA256 so the skeptic adjudicates against the same bytes the investigator saw — not a paraphrase.

Decoupling from infrastructure mattered under a three-day clock. The whole loop is testable offline against captured fixtures via the local claude CLI, so the software spine could be built and measured without waiting on a live SIFT VM.

Accomplishments

• A working finding-confirmation loop with a measured, third-party-benchmark accuracy number.
• Read-only enforced at two independent architectural layers, exercised by tests, rather than asserted in a prompt.
• A provenance chain that is simultaneously the evidence-integrity mechanism and the submission's audit-trail deliverable.
• 32 passing tests and an offline demo that needs neither a SIFT box nor an API key, covering the confirm / retract / drop paths end-to-end.

What we learned

Eliminating hallucinated findings while keeping the grounded ones — not maximizing raw recall — is the capability the IR-accuracy and audit-trail criteria actually reward. A precision-first skeptic that drives FP-rate to zero, paired with corroboration to recover the true positives it conservatively drops, fits that incentive better than a high-recall report-everything baseline.

The recall cost is real and we left it visible: on the external set the skeptic marked 2 of 9 true positives ungrounded. That is the price of precision-first adjudication, and it is exactly what cross-tool corroboration exists to claw back (not exercised in the single-pass measurement above).

What's next

• Tool-execution-on-a-real-image evaluation — running Volatility et al. against an actual memory/disk capture with knowable ground truth. DFIR-Metric ships no images, so this requires the live forensic toolchain and is in progress.
• End-to-end recall — the numbers above measure the confirmation layer over a fixed draft set, not whether the investigator locates every artifact in an image. Closing that loop is the next measurement.
• Statistical robustness — small n (9 external + 16 controlled), single run each. Broadening the external set beyond single-IP log-analysis challenges and running repeated trials.
• Live SIFT + Protocol SIFT integration — registering the MCP server with Claude Code on a real SIFT Workstation and running the agent against mounted evidence.

License

MIT.

Built With

• python