Inspiration
Most "AI DFIR" demos look impressive until you ask one question: can you prove that? Tools like Protocol SIFT wrap forensic CLIs with permission blocklists and prompt guardrails - but the model can still be given a shell if someone misconfigures it, and nothing forces every finding to be independently re-checked against raw evidence.
Anthropic's GTG-1002 disclosure made the stakes concrete: attackers ran Claude Code + MCP at high autonomy. I wanted the defender-side mirror - same MCP-shaped boundary, pointed at finding evil instead of doing it.
The FIND EVIL! hackathon was the forcing function: one command, zero human input, architectural constraints judges can actually verify, and an honest accuracy score against published ground truth.
What it does
VERDICT is an autonomous DFIR agent for the SANS SIFT Workstation:
verdict investigate <case-folder>
Walk away. It surveys evidence, triages a Windows intrusion kill chain (initial access → persistence → lateral movement → C2), runs an adversarial verify pass on every finding, and emits a cited HTML/PDF report plus an append-only ledger.jsonl the agent cannot edit.
Every finding gets a confidence state: VERIFIED, UNCONFIRMED, or REFUTED. Overclaims are flipped on screen - the smoke-case centerpiece is a file named mimikatz.exe that triage flags as credential theft, then verify REFUTES when the content is 12 bytes of plain text (hello world), not a PE executable.
One command. Thirteen typed read-only MCP tools. No shell. Evidence read-only. Ledger server-written. Budget guard at $5/run.
Try the ~3-minute smoke case in the repo, or the full ~30 GB Szechuan Sauce case for the accuracy report path.
How I built it
Two processes, one hard boundary:
Orchestrator (verdict/) - Typer CLI, manual Anthropic agent loop (deliberately not the Claude Agent SDK, which ships bash/file tools you must disable), triage + per-finding verifier with fresh context and phase-specific tool allowlists, Jinja2 report generator (HTML + PDF).
MCP server (verdict_mcp/) - Pydantic validation → path guard (case folder read-only, run folder write-only) → fixed-binary subprocess runner (shell=False, timeouts, output caps) → server-only append-only ledger. Backends: Sleuth Kit, Volatility 3, YARA, EvtxECmd / EZ Tools fallbacks.
Key design choices:
- Manual loop vs Agent SDK — Harmful tools are absent from API requests, not permission-blocked.
- Triage + verify split — Triage optimizes recall; precision comes from reproduction, not prompt pleading.
- Bodyfile/mactime vs Plaso — Full supertimeline on 30 GB images breaks the ~30-minute target.
- Smoke case before Szechuan — Pennies-before-terabytes; caught real parser bugs stubs never surfaced. Architecture diagram: docs/architecture.md
Challenges I ran into
Real binaries ≠ stub tests. EvtxECmd writes UTF-8 BOM and nested JSON; probe argv leaked into real calls once; SHA replay drift hashed run-stamped logs instead of deterministic records. Each bug appeared only on real smoke runs.
DC01 disk layout. The 11 GB DC E01 did not yield a walkable C: at naive offsets. Early Szechuan runs burned budget on empty fs_list until I added mmls partition auto-discovery. Desktop disk + memory remained the reliable path when DC C: stayed partial.
Volatility 3 on Server 2012 R2 DC memory. Symbol packs, long runtimes, system vol vs venv volatility3 - DC memory stayed best-effort; desktop memory was the blocker test.
Making REFUTED reproducible. Smart triage sometimes reads the decoy early and hedges ("filename only"). I tuned triage for recall on scary names and let verify knock down capability claims so the demo flip stays deterministic.
$5/run vs depth. Full Szechuan runs take ~30–60 minutes and ~$3–4 API spend. Image tools, per-finding verifier passes, and prompt caching all compete for the same budget guard.
Accomplishments that I'm proud of
Architectural safety that is judgeable - no bash, delete, or arbitrary-write tools exist in the 13-tool surface; path guard blocks writes to evidence; refusals land in ledger.jsonl as tool_rejected. The REFUTED flip on demand - smoke decoy reliably goes from critical T1003.001 overclaim to REFUTED with a one-line reason and full refutation in Appendix A. 100% precision on the primary Szechuan run - 12 findings, all VERIFIED; zero hallucinated claims reached the report (~40% recall against 24 published ground-truth facts, with every miss named honestly). Disk ↔ memory correlation - coreupdater.exe correlated across filesystem/timeline and Volatility process list (F-004), hitting the case's headline challenge pattern. Full audit trail - every tool call, args, output SHA-256, and citation click-through in the report; committed execution logs for judges. Shipped all 8 submission components - repo, demo path, architecture diagram, dataset docs, accuracy report, try-it instructions, and execution logs. What I learned Safety belongs in architecture, not prompts. Asking the model to behave is not the same as removing the capability to misbehave. Verification changes the product story. Triage recall + verifier precision beats a single pass tuned for scary output; REFUTED findings belong in the appendix, not silent drops. Honest accuracy beats inflated scores. A credible ~40% with named misses and 100% precision is stronger than a suspicious 100%. MCP is the right shape for DFIR when the server owns paths, binaries, and the audit log - the model gets hypotheses and 8 KB excerpts, not the keys to the evidence room. Pennies before terabytes. A 3-minute smoke case saved hours of expensive debugging on the 30 GB dataset.
What's next for VERDICT - Autonomous DFIR Agent That Proves Its Findings
Deep pcap parsing - close GT-12 URL/host gaps (IE download, C2 IPs) without manual analyst work. Stronger DC disk access - partition-aware bodyfile cache; NTDS/Share artifacts when C: is reachable at the correct offset. Resume interrupted runs - checkpoint/resume would save budget on API outages mid-investigation. Cross-run IOC memory - interesting for SOC workflows, out of scope for a one-case MVP. Opus verifier option - quality ceiling vs $5 cap tradeoff as models get cheaper. Long term: a trustworthy autonomous triage layer IR teams can actually act on - every finding cited, every action audited, zero hallucinated evil.
Log in or sign up for Devpost to join the conversation.