What we learned## Inspiration
AI-enabled attackers now move at machine speed, but incident response is still a human reading logs one line at a time. The bottleneck isn't the tools — SANS SIFT already ships world-class forensics — it's triage speed across a multi-host intrusion. The obvious fix, "point an LLM at the evidence," fails the one test that matters in forensics: it hallucinates, and it can modify evidence. We set out to build an agent that triages at machine speed and that a practitioner could actually stand behind.
What it does
An autonomous DFIR agent that triages a multi-host intrusion (disk .E01 + memory), reconstructs the full attack chain, and produces a fully-cited forensic report — without hallucinating and without ever being able to modify evidence.
- Runs the whole pipeline itself: manifest → hash → read-only mount → memory + disk + timeline → cross-host correlation → report.
- Every finding cites the exact artifact (tool + command +
provenance_id) it came from, or a citation linter drops it. - Self-corrects: reconciles disk↔memory contradictions — e.g. disputing a benign service that memory flagged as C2.
- Cannot spoliate evidence — the architecture exposes no write/exec tool.
How we built it
The core idea: code decides, the LLM only narrates.
- A custom MCP server wraps SANS SIFT's court-vetted tools (Volatility 3, Plaso, The Sleuth Kit, EZ Tools, bulk_extractor) as 28 typed, read-only functions. No shell, no write-to-evidence — the menu of tools is the security boundary.
- Detection, correlation, dedup, confidence and contradiction checks are deterministic Python rules — never model judgment.
- Two path-validated roots — read-only
EVIDENCE_ROOTvs write-onlyCASE_ROOT; every action (success, failure, refusal) appends to an immutable provenance ledger.
Accuracy (measured, not claimed)
We ground-truthed the SANS SRL-2015 case into a 10-milestone, evidence-adjudicated oracle, then scored deterministically:
| Stock baseline | This agent | |
|---|---|---|
| Recall | 0.90 | 1.00 (10/10) |
| Hallucinations | ~1 / run | 0 |
| Consistency | nondeterministic | same 10/10 every run |
1,244 logged tool executions; the full chain (Java drive-by → httppump/spinlock implants → PsExec lateral movement → DC compromise → RAR exfil) recovered and individually cited.
Challenges we ran into
During a from-scratch submission run, the agent stalled and filled the disk — a concurrency bug in the timeline node (a 1.6 GB export re-run per pivot, then two racing on the same file). We diagnosed it, fixed it (export once per host, guarded by a lock), and hardened it under test. Finding and fixing a real failure mid-build is exactly the self-correction story the agent is about. We document this and other honest gaps (a low-precision shared-binary heuristic; an un-run live spoliation test) in our Accuracy Report — a report that claims no weaknesses isn't credible.
What we learned
- Anti-hallucination must be architectural, not a prompt: make the unsafe action impossible to invoke, not just discouraged.
- On a real case, "accuracy" isn't one number — recall is measurable against ground truth, precision on novel findings isn't, so verifiability (every claim traces to raw evidence) matters more than a score.
What's next
The engine is OS-agnostic (analyzer routing for Windows/Linux/macOS/network devices already scaffolded). Next: live/triage data (Velociraptor/EDR) and lighting up Linux, macOS, and network-device agents — same read-only, cited, self-correcting guarantees — for complete enterprise coverage.
Log in or sign up for Devpost to join the conversation.