-
-
Overview
-
Evidence Graph
-
Report Summary
-
Terminal Proof
-
Submission Package
Inspiration
Modern incident response is a race against time. Early triage often decides whether a responder catches persistence quickly or spends the first critical minutes setting up tools, checking paths, and validating noisy evidence by hand.
LLMs can help move faster, but in DFIR they introduce two dangerous failure modes: they can overclaim based on weak signals, and they can interact with evidence in unsafe ways if given broad shell access.
EvidenceGuard SIFT was built to solve that gap: an autonomous, read-only DFIR triage agent that works inside a SANS SIFT-style workflow, correlates evidence across artifacts, and produces findings that are traceable, reviewable, and benchmarked against ground truth.
What it does
EvidenceGuard SIFT analyzes a synthetic Linux filesystem case and identifies evidence-linked findings across:
- persistence artifacts such as
systemdservices and cron jobs - suspicious high-risk filesystem paths
- user activity artifacts such as shell history
- SHA-256 hashes and direct path references
- audit logs for every tool execution
Instead of declaring a finding confirmed from a single clue, EvidenceGuard uses an Evidence Graph. A finding is only categorized as CONFIRMED when independent artifacts support it, such as a persistence file referencing a high-risk executable and shell history also referencing that same path.
Lower-confidence results stay as INFERENCE, with explicit missing evidence listed for human review.
How we built it
We built EvidenceGuard SIFT as a Python package with a strict read-only forensic workflow.
The core pieces are:
- a CLI for initializing a synthetic case, running analysis, benchmarking, and generating reports
- structured tools for filesystem, persistence, suspicious path, and user activity scanning
- Pydantic schemas for findings, evidence details, confidence rules, and audit metadata
- a cryptographic audit trail where tool outputs are hashed with SHA-256
- a benchmark harness that compares findings against
benchmarks/sample_ground_truth.json - Markdown and JSON report generation for judge and analyst review
The benchmark checks whether expected findings are detected, whether basename-only matches stay as inferences, whether single-source findings avoid overclaiming, and whether forbidden speculative wording appears in confirmed findings.
Challenges we ran into
The hardest part was forcing the system to avoid being “too confident.”
A file named evil.sh is intentionally obvious in the sample case, but the agent still should not jump to a conclusion from the filename alone. It needs evidence linkage: a path reference, a matching hash, or supporting user activity.
We also had to make the reports judge-ready. Every finding needed exact paths, SHA-256 hashes, confirmation rules, evidence strength, tool call IDs, output hashes, and missing evidence when something remained an inference.
What we learned
LLMs are useful in DFIR only when constrained by deterministic tools, evidence schemas, and auditability. The winning pattern was not “let the model investigate freely.” It was “force every claim through evidence correlation and make every step inspectable.”
We also learned that benchmark-driven development is extremely effective for reducing overclaiming. The project now has a repeatable synthetic accuracy benchmark that scores the agent’s findings against known ground truth.
Accomplishments
- Built a complete vertical slice for autonomous Linux filesystem triage.
- Added read-only evidence handling and path safety checks.
- Implemented evidence-linked confirmation rules.
- Added SHA-256 audit trails for tool outputs.
- Built a ground-truth benchmark harness.
- Achieved 13/13 passing tests.
- Achieved 100/100 on the synthetic benchmark sample case.
- Produced judge-facing documentation, runbook, release checklist, and final reports.
What’s next
The current version focuses on filesystem, persistence, suspicious paths, and user activity. Future work would extend the same evidence-correlation framework to memory analysis, network artifacts, SSH artifacts, Windows event logs, and timeline reconstruction — without weakening the read-only and overclaim-resistant design.
Log in or sign up for Devpost to join the conversation.