Council-SIFT

Council-SIFT: adversarial review loop for autonomous DFIR findings.

Inspiration

Autonomous digital-forensics agents are useful only if their findings survive contact with the evidence. In incident response, a confident but unsupported claim can waste an investigation or mislead a human examiner. Council-SIFT was inspired by that gap: agents can run tools, but they also need an adversarial verification layer that asks, “What does the evidence actually prove?”

What it does

Council-SIFT is a verification layer for autonomous digital forensics and incident response. An analyst agent runs SANS SIFT-style forensic tools, drafts findings, and submits each finding to a Council. The Council checks every claim against the actual tool output before a human is asked to trust it.

If the claim is unsupported, the Council bounces it back and the agent self-corrects. If the claim survives review, the system emits receipts, hashes, logs, and trace links so a human can audit exactly what happened.

The demo example is a Rar.exe process found in SRL-2018 memory evidence. A bad finding overclaims that Rar.exe proves exfiltration to attacker C2. Council-SIFT rejects that: a process listing only proves the process existed. The corrected finding says Rar.exe is a staging indicator that needs disk/timeline correlation, not proof of exfiltration by itself.

How we built it

We built Council-SIFT around a SANS SIFT-style workflow. The analyst agent uses forensic tools such as Sleuth Kit, Volatility 3, Plaso/log2timeline, and psort. Commands pass through a default-deny identity kernel that blocks destructive actions and preserves read-only evidence boundaries.

Findings are stored in a Neo4j-backed MCP memory and review layer. Five deterministic Council seats check citation grounding, tool semantics, contradiction, inference boundaries, and scope boundaries. A synthesis step decides whether the finding is verified or bounced. Verified findings receive receipts and audit trails; bounced findings become self-correction records.

The repo includes deterministic no-key replay scripts so judges can reproduce the core loop without credentials, plus logs from genuine autonomous runs.

Challenges we ran into

Forensic claims are easy to overstate. A process listing can show that Rar.exe existed, but it cannot prove C2, exfiltration, intent, or actor identity by itself. We had to make those inference boundaries explicit.

We also had to make evidence safety architectural, not prompt-only. The system refuses destructive commands before execution and treats official evidence as read-only.

Finally, reproducibility is hard because official forensic evidence cannot be redistributed. We separated judge-runnable no-key checks from full official-evidence replay, which requires SANS SIFT plus the organizer datasets.

Accomplishments

We demonstrated a complete self-correction loop on real SRL-2018 evidence: an unsupported Rar.exe exfiltration overclaim was bounced by the Council, narrowed by the agent, and re-verified as a staging indicator rather than proof of exfiltration. The public repo includes logs from 9 genuine autonomous runs across disk, memory, and telemetry, with 46 drafted findings, 36 verified findings, and 10 self-corrections.

On the scoped injected unsupported-claim benchmark, Council-OFF lets 85/85 bad claims reach review, while Council-ON catches 85/85. A separate blind red-team rescore reports 98.6% recall and 98.6% precision, with one false positive and one miss disclosed.

What we learned

Autonomy in digital forensics is not just tool use. It also requires disciplined refusal: knowing what a tool output does not prove. The useful agent is not the one that writes the most confident report; it is the one whose claims can survive contact with the evidence.

What’s next

Next we would package a cleaner native SIFT/OVA path, expand endpoint and log-source coverage beyond the current disk, memory, timeline, and telemetry examples, add more held-out external benchmarks, and continue measuring the additive LLM skeptic panel separately from the deterministic Council floor.