EvilTrace

## Inspiration

In November 2025, Anthropic disclosed GTG-1002 — a state-sponsored operation that used Claude Code to run reconnaissance, exploitation, and lateral movement at 80–90% autonomy, at request rates the team called "physically impossible" for humans. That was offense. On defense, responders still look up command-line flags during live incidents, and Protocol SIFT — while powerful — hallucinates more than we'd like. We wanted a defender that responds in seconds without overclaiming: an agent that, like a senior analyst, recognizes when something doesn't add up and corrects itself.

## What it does

EvilTrace is a self-correcting, evidence-grounded DFIR agent for Protocol SIFT on the SANS SIFT Workstation. All evidence access goes through a typed MCP server — there is no execute_shell tool, so the agent physically cannot run destructive commands. It registers and hashes evidence, plans an investigation, runs read-only forensic tools (tshark, Sleuth Kit, Volatility), normalizes results into an evidence graph, proposes candidate findings, and validates each one against provenance, manifest hashes, corroboration count, contradictions, and hallucination (entity-not-in-graph) checks. Unsupported claims are rejected, overconfident ones are downgraded, and the loop re-plans until a bounded --max-iterations cap. Every final finding traces back to a tool execution and the original evidence SHA-256. It is decision-support and triage for an investigator — not a court-ready system.

## How we built it

We chose the FIND EVIL "Custom MCP Server" pattern (the one the hackathon calls the most sound architecture). The system has three layers: a typed MCP evidence layer (structured forensic functions + a command allowlist/denylist + read-only evidence paths + timeouts/output caps), a validation/reporting layer (a finding schema with confirmed/inferred/rejected/needs_review, contradiction and hallucination detectors, a machine-comparable benchmark), and a bounded self-correction loop. Two drivers sit in front of the same typed tools: a deterministic reference orchestrator (eviltrace run, fully reproducible, ~0.5s per investigation, zero LLM tokens) and Claude Code headless. Every tool call emits a JSONL audit event plus a provenance.schema.json record (command, exit code, durations, stdout/stderr hashes). We validated the whole thing on the real SIFT Workstation (Ubuntu 24.04, native tshark/Volatility/Sleuth Kit).

## Challenges we ran into

The hardest problem was stopping the agent from overclaiming architecturally, not just with prompts. Prompt rules can be ignored, so we made the MCP server expose no shell, made evidence writes raise a guardrail error, and built a validation engine that can reject or downgrade a finding regardless of what the model says. The second challenge was making self-correction genuinely visible rather than a single staged reject — we designed the run so the same finding goes confirmed → downgraded to inferred → targeted re-plan → upgraded back to confirmed across two iterations, with the full trace preserved. The third was making it actually install and run on SIFT (a Python 3.10 floor from the MCP SDK, Volatility exposed as vol rather than volatility3, no uv preinstalled) while keeping the audit trail complete (the real tshark raw outputs ship in the repo).

## Accomplishments that we're proud of

• A typed-MCP trust boundary with no execute_shell — destructive actions are architecturally impossible, and we tested for spoliation (writes to cases/ are blocked; before/after evidence hashes are identical).
• Genuine multi-iteration self-correction with demonstrable improvement between the first and final iteration on the same evidence.
• Complete provenance: every finding → audit_id → the exact tool command → evidence SHA-256, with real tshark commands recorded.
• Validated on the real SANS SIFT Workstation in ~0.5 seconds per investigation, with a machine-comparable known-answer benchmark (artifact recall 1.0, hallucination rate 0.0 on the bundled sample).
• 49 automated tests; every generated artifact validates against its JSON schema.

## What we learned

Autonomous DFIR is less about giving the model more tools and more about constraining how tools are used. The single most important pattern was separating hypothesis generation from evidence validation: the agent may propose anything, but the system only finalizes findings that survive provenance, manifest-hash, contradiction, and corroboration checks. We also learned that reproducibility matters as much as autonomy — a deterministic mode that produces the identical, fully-traced result on every run is what makes the accuracy claims checkable.

## What's next for EvilTrace

• Validate the disk, Windows, and memory typed wrappers against NIST known-answer cases (they are implemented and degrade gracefully today).
• Measure EvilTrace against the Protocol SIFT baseline directly — fewer hallucinated findings on the same data is the hackathon's success metric.
• Multi-source correlation: cross-reference a disk image against a memory capture and flag discrepancies.
• More machine-comparable ground-truth sets, and OpenSearch indexing for large cases.

Built With

• protocol-sift
• sans-sift-workstation
• sleuthkit
• virtualbox
• volatility3
• wireshark