Inspiration
I come from a data science competition background. In KDD Cup 2026 (Tencent UNI-REC Challenge, Industrial Track), I placed 269th out of 689 participants — not bad for my first attempt using AI as a development partner. That experience taught me something important: AI doesn't replace rigorous thinking. It amplifies it. The ones who win aren't the ones who write the most code. They're the ones who know how to test, verify, and iterate.
When I saw FIND EVIL! and learned that Protocol SIFT "hallucinates more than we'd like," I recognized the exact same problem I'd spent months solving: how do you trust what the model tells you? My KDD Cup experience was all about building systems that produce reliable results. This project applies that same experimental mindset to DFIR — treating AI hallucinations not as a fatal flaw, but as a testable, measurable, fixable problem.
What it does
This project extends Protocol SIFT with a self-correction loop. After the agent completes its initial triage on a disk image, it automatically re-examines its own findings — checking for hallucinated files, timeline inconsistencies, and missed artifacts — then adjusts its conclusions before generating the final report. The goal is not to replace the security analyst, but to make the AI analyst honest enough to say "I'm not sure, let me check that again."
How we built it
- Started with the baseline Protocol SIFT agent on the SIFT Workstation.
- Designed a self-correction prompt chain that forces the agent to verify key claims (file existence, timeline logic, coverage gaps).
- Built an automated testing harness using ground-truth case data to measure accuracy improvement across iterations.
- Used Claude Code and DeepSeek as primary development assistants for both code generation and prompt iteration.
Challenges we ran into
- Balancing thoroughness with speed: too many self-check iterations slow down triage, too few leave hallucinations uncaught.
- Designing verification prompts that catch errors without introducing new ones.
- Building a meaningful test framework from scratch as someone new to DFIR concepts.
- The agent sometimes "double downs" on a hallucination when asked to verify — teaching it to genuinely self-correct rather than justify its mistakes was harder than expected.
Accomplishments that we're proud of
- Built a working self-correction loop that measurably reduces hallucinations compared to the Protocol SIFT baseline.
- Created a reproducible accuracy benchmark so any future improvement can be quantitatively evaluated.
- Shipped a complete, working project in 21 days while learning both the SIFT toolchain and DFIR fundamentals from scratch.
- Approached the problem with an experimental mindset (control variables, baseline comparison, iterative testing) rather than treating prompt engineering as guesswork.
What we learned
- Prompt engineering is experimental science — every change needs to be tested against a baseline, just like model training.
- Self-correction works best when the agent is forced to cite specific tool output for each claim, not just reason in the abstract.
- A rigorous testing mindset can meaningfully improve AI reliability even without deep domain expertise.
- The DFIR community has built incredible open-source tools; making AI work reliably with them is a worthy engineering challenge.
What's next for SIFT Self-Correction Agent
- Expand the test suite with more diverse and complex case data.
- Explore the Custom MCP Server architecture to enforce evidence integrity at the architectural level, not just through prompt guardrails.
- Experiment with multi-agent decomposition: one agent analyzes, another independently verifies.
- Open-source everything so the community can build on it and contribute better test cases.
Log in or sign up for Devpost to join the conversation.