Agentropix-SIFT

Tagline: Autonomous DFIR triage on the SANS SIFT Workstation that finds evil without ever letting the LLM author, rate, or seal a single finding — deterministic tools find it, a human signs it.

Built With: python, asyncio, fastmcp, model-context-protocol, claude, anthropic, pydantic, volatility3, sleuthkit, plaso, log2timeline, yara, bulk-extractor, regripper, evtx, ewf-tools, ez-tools, libpff, oletools, pytsk3, sha-256, hmac, pbkdf2, defusedxml, tenacity, psutil, pytest, wazuh, opensearch, jinja2, weasyprint, typer, rich, hatchling, uv, sans-sift-workstation, tailscale

Inspiration

DFIR triage is the bottleneck. When an image lands on an examiner's desk — a Windows disk, a memory dump, a domain controller's C: — the first hours go to the same mechanical sweep: list the processes, walk the filesystem, scan for injected code, parse the event logs, hunt the IOCs. It is exactly the repetitive first pass an agent should accelerate. So the obvious move is to point an LLM at the evidence and ask it what it finds.

That is also the one thing you must never do.

The moment an LLM authors a forensic finding, you have wired a fabrication risk straight into a chain of custody. A model that hallucinates a PID, invents a registry key, or confidently rates its own guess at 0.9 isn't a triage accelerator — it's a liability that gets thrown out in court. The whole field's instinct, correctly, is to distrust it.

Our thesis is that you can have autonomous triage AND evidentiary integrity — but only if the LLM never touches a fact. Agentropix-SIFT is built around that single load-bearing constraint: LLM at the edge, determinism inside. The model can propose which agents to run and narrate the result in plain English, but every recorded finding is produced by a deterministic forensic binary — Volatility3, Sleuth Kit, YARA, EVTX, RegRipper — captured at the MCP boundary and fingerprinted with SHA-256. The control loop that decides when triage is done (the Trinity Loop's Critic) halts on a deterministic convergence fingerprint, never on a model's self-assessed confidence. There is, by design, no write tool anywhere in the 73-tool surface — the agent structurally cannot alter the evidence it reads.

We leaned hardest into the part most demos hide: honest negatives are the feature. Our memory recall is 108/118 (91.5%) — and we report the 10 it misses rather than rounding up. On the VANKO insider-IP-theft case, the engine confirmed 10 findings of 19 hypotheses and let a false-positive gate refute the other 9 — keeping every refuted hypothesis in the report as an honest negative instead of quietly dropping it. A triage agent you can check is worth more than one you're asked to believe. The examiner stays the examiner of record, and the only way a finding leaves DRAFT is a human HMAC sign-off — the one hard-stop a model can't reach around.

That is what we wanted to bring to "Find Evil!": an agent that genuinely finds the evil — fast, and on the local host — engineered so you can trust why it says what it says.

What it does

Agentropix-SIFT is an autonomous DFIR triage agent that runs on the SANS SIFT Workstation and is exposed to Claude as one governed MCP server. Point it at a Windows disk or memory image and it performs the first pass an examiner would otherwise do by hand — process and network triage, timeline building, registry and event-log analysis, IOC hunting, mailbox carving — and hands back a cryptographically sealed, schema-validated triage report. The whole forensic path runs fully local on the host: no API keys, no cloud dependency for the evidence work.

The design principle is load-bearing and we never relax it: the LLM never touches a fact. Every finding is produced by a deterministic forensic binary, fingerprinted with SHA-256, and tagged with provenance — precisely so a human can verify it. The model proposes and narrates; it never authors a finding and is never in the halt path.

What it ingests

Disk images — EWF/E01 and raw .dd, read-only.
Memory images — raw .mem / .raw / .bin dumps.
Windows artifacts off those images — the registry, .evtx/.evt event logs, Amcache, and (for insider cases) Outlook OST/PST mailboxes.

Evidence is structurally read-only: a deny-by-default Thymus policy checks every call against a path allowlist before any subprocess spawns, and there is no write tool in the surface at all — the engine reads and SHA-256-fingerprints the image, but cannot mutate it.

The Trinity loop — how it triages

A deterministic three-part control loop drives the investigation:

Architect — proposes which agents to run, returning the canonical swarm in priority order and (via a default-on Reflexion-lite step) pruning agents the Critic already marked stable.
Swarm — a 7-agent swarm of DFIR specialists (Memory, Timeline, Filesystem, Artifact, Discovery, Mail, Hunt), interleaved with six deterministic MITRE ATT&CK detectors (13 SwarmAgent classes in total). Each agent drives real forensic tools and writes Findings to a shared blackboard; an observation is promoted to a Correlation only when enough agents corroborate the same token (quorum default 2).
Critic — scores the accumulated findings with a closed-form rule (score = min(1.0, max_confidence + 0.25 · len(correlations))) and halts on a deterministic convergence fingerprint — when the findings stop changing, bounded by a hard iteration budget — never on an LLM's self-assessed confidence (halt default 0.85, trinity/critic.py).

The tool surface

One FastMCP server exposes 73 distinct MCP tools over stdio and HTTP. Of those, 16 are SIFT forensic wrappers — hardened drivers (timeout, memory ceiling, retry, stderr capture, tracing) around the trusted SIFT binaries (Volatility3, Plaso, Sleuth Kit, EVTX, YARA, bulk_extractor, RegRipper, EZ-Tools, mail parsers). The remaining tools cover case lifecycle, finding records, reporting, provenance, the executable-artifact registry, HMAC examiner approval, and the Wazuh push. Crucially, the same @app.tool() functions the swarm calls are the ones an LLM client calls — the model is just another consumer of the tool surface, not wired into the findings.

What it produces

Findings, each tied back to the exact tool output via raw_stdout_sha256 and tagged inference_constraint = high.
A human-in-the-loop gate: findings stay in DRAFT until an examiner signs off through the HMAC approval sidecar (PBKDF2, nonce, append-only approval hash chain). This is the one place a person — not a model — must act before anything is sealed.
A Courtroom seal: SHA-256 evidence byte-binding plus an HMAC-SHA256 seal over the report and audit log, giving a tamper-evident chain of custody a judge can independently verify.
An optional Wazuh/SIEM push of already-approved findings and IOCs, behind default-deny kill switches and an active-response CIDR guard.

What it has actually found (real cases)

We ran it end-to-end on three sealed cases. The numbers below are the real results, including the honest negatives — refuting a hypothesis is a feature here, not a failure:

SRL-2018 — a multi-host APT IP-theft scenario across the Stark Research Labs estate (initial access → credential theft → lateral movement → Cobalt Strike/Empire C2 → collection & exfiltration). A real engine run over the domain-controller E01 (5 iterations, 13-agent swarm, HMAC-sealed) produced 10 examiner-approved findings, with nine recovered-malware SHA-256 hashes published (the bytes kept out of the repo).
SRL-2015 — a four-host APT enterprise case (4 hosts × disk + memory = 8 sealed runs, 15-iteration loops). The swarm produced 2,233 raw findings, distilled to 17 examiner-approved findings; of a 91-IOC set, threat-intel enrichment flagged 12 as malicious, and 21 malware samples were recovered and hash-verified — including the cross-host spinlock.exe → Domain-Controller correlation.
VANKO ("The Case of the Abducted Zebrafish") — an insider IP-theft case (deliberately not a malware intrusion): a trusted insider archiving classified data and exfiltrating it over dual cloud channels. Of 19 hypotheses, 10 were confirmed and 9 were refuted by the false-positive gate — the agent reported the honest negatives rather than inflating the count, which is exactly the discipline DFIR demands.

Recall is reported the same honest way: 72/72 (100%) on disk regression and 108/118 (91.5%) on memory — we publish the misses, because a triage accelerator earns trust by being checkable, not by claiming to be infallible. You remain the examiner of record.

How we built it

The whole design follows one rule: LLM at the edge, determinism inside. The model picks what to look at; everything that touches evidence is deterministic, code-enforced, and provable in court. That principle decided every technology choice below.

Custom MCP Server (FastMCP)

Agentropix-SIFT is a purpose-built FastMCP 3.2.4 server (FastMCP("agentropix-sift")) that exposes its 73 forensic tools to any general-purpose MCP client — Claude Code, Claude Desktop (via an npx mcp-remote stdio shim), or a local stdio launch. The 73 tools are 68 @app.tool() registrations in mcp_server/fastmcp_app.py plus 5 from the Wazuh registrars. It is deliberately not a Claude plugin and not an agentic IDE: it is a standalone server you mount into a session, and it ships no LLM client of its own in the MCP path.

Two transports: stdio is the default (trust = process UID); streamable HTTP is opt-in on port 8765 at /mcp, loopback-only unless explicitly exposed (tailnet-only per ADR-017). The HTTP path is fail-closed: _build_app() raises and refuses to start without AGENTROPIX_MCP_AUTH_TOKEN (unless dev mode is set explicitly). Bearer auth uses secrets.compare_digest for a constant-time compare, and every /mcp request is written to a JSON-lines audit log (/var/log/agentropix/http_audit.log) where the token is recorded only as sha256[:16], never in the clear.

The server is the enforcement boundary. The shared tool core (mcp_server/server.py, 61 mcp_* async functions) carries the in-code statement: "The MCP server is the enforcement boundary — Thymus policy runs here, not in the agent." In front of every wrapper sit a per-tool sliding-window rate limiter (default 60/min) and 52 Thymus check_read call sites.

Deterministic Swarm — async coroutines, no LLM coupling

Inside the package runs the Trinity Loop + DFIR Swarm: 13 SwarmAgent classes (7 core specialists — Memory, Timeline, Filesystem, Artifact, Discovery, Mail, Hunt — plus 6 ATT&CK detector agents) coordinating over an asyncio-locked Blackboard. The source is explicit that this is LLM-free: agents are "pure async coroutines over the MCP boundary — no LLM coupling" (agents/_base.py docstring). The Blackboard surfaces cross-source correlations (filenames, hashes, IPs, PIDs appearing across ≥ a quorum of agents, default 2), and a deterministic Critic decides halt-vs-iterate as a pure function of Blackboard state (max finding confidence + 0.25·#correlations, halt threshold 0.85, with a guard that refuses to halt while any planned agent produced zero findings). The Swarm calls the tool core by direct Python import, not network MCP, so the same deterministic primitives back both the agentic CLI and the MCP surface.

The only in-runtime LLM touchpoint in the entire package is an optional, default-off, fail-open Architect reorder pass (AGENTROPIX_ARCHITECT_LLM_REORDER, which lazy-imports the anthropic SDK) — and even that is code-validated: the reordered agent set must exactly equal the deterministic set or it falls through to deterministic order. The model gets a vote; the code keeps the veto.

Forensic wrappers around the SIFT binaries

Every fact the agent reports originates from a named deterministic tool wrapping one of 16 SIFT Workstation binaries: Volatility 3 (vol), log2timeline/Plaso (log2timeline.py), Sleuth Kit (fls/icat/mmls), ewf-tools (ewfinfo), EVTX (evtx_dump.py), YARA, bulk_extractor, RegRipper (rip.pl), pf, amcache_parser, shimcache_parser, exiftool, foremost, and hashdeep — with the Eric Zimmerman tools (MFTECmd, RECmd, LECmd, JLECmd, SBECmd, SQLECmd, bstrings) run as genuine .NET binaries via dotnet, and oletools maldoc analysis as an in-process Python library.

The invocation discipline is strict (mcp_server/wrappers/_subprocess.py): everything runs through asyncio.create_subprocess_exec with an argv list — never a shell, eliminating shell-injection as a class of bug. Seven memory-heavy wrappers (Volatility, Plaso, bstrings, JLECmd, SBECmd, SQLECmd, PDF text extraction) get an RSS memory cap scaled to image size; the rest are timeout-kill only. Wrapper response models carry the SHA-256 of the actual subprocess stdout (raw_stdout_sha256), so every parsed result is grounded to the raw bytes the tool actually emitted.

Thymus read-only evidence gate

The Thymus policy (mcp_server/thymus_policy.py) is the architectural backbone of evidence integrity. It allowlists evidence prefixes (/cases/, /mnt/, /media/, /evidence/, /tmp/agentropix-sift-*), forbids traversal patterns (.., ~, /dev, /proc, /sys), and defends against URL-decode, symlink, and PATH_MAX tricks. Crucially, check_write() on evidence unconditionally rejects — the agent literally has no tool to write to evidence. Derived output is the only thing it can produce, and only into allowlisted out-dirs.

HMAC examiner-approval hard-stop (human-in-the-loop)

This is the control that earns the project's name. Every finding the agent ingests is force-stamped DRAFT by the W-286 gate, which strips any caller-supplied approval.* field — the LLM cannot self-approve through any write surface. Promotion from DRAFT to APPROVED happens only through the Examiner Approval Portal (a Starlette service on port 8800): a POST /challenge issues a 60-second single-use nonce; POST /approve requires an HMAC-SHA256 signature with a PBKDF2-600k-derived key from the examiner's password, bound to (examiner, target), consumed even on failure to block replay. A precondition gate verifies the target exists and currently holds the asserted status before any approval is written, and a per-case prev_approval_hash chain makes the approval ledger tamper-evident. Reports (report_generate) reconcile APPROVED-only findings and warn explicitly when there are 0 APPROVED but DRAFTs exist. No code path writes an approval without a valid signature.

Anti-hallucination and provenance grounding

Because the agent only reports what deterministic tools emit, hallucination is structurally constrained rather than merely discouraged. Final reports are sealed with HMAC-SHA256 using a per-run 32-byte session key; the audit-log seal is cross-bound into the report before the report seal, and evidence_image_sha256 binds the report to the image bytes. Re-verification CLIs (audit/verify_seal.py, provenance/validate.py) are hard gates that exit non-zero on any forged or malformed row. We keep an honest-negatives discipline throughout — memory recall is 108/118 (91.5%), not 100%, and refuted hypotheses stay in the record as honest negatives rather than being quietly dropped.

Architectural vs prompt-based guardrails

We draw a hard line that matters in court: an architectural guardrail is enforced by code the model cannot reach around; a prompt-based guardrail is an instruction the model is expected — but not forced — to honor. The load-bearing controls above are all architectural: the Thymus read-only gate, fail-closed Bearer auth, the rate limiter, the _safe_tool error envelope, the W-286 DRAFT gate, the HMAC approval sidecar with nonce anti-replay, single-use scoped egt_ mutation tokens, the WAZUH_DRY_RUN_ONLY kill switch, false-positive denylists, courtroom HMAC sealing, and the deterministic Critic. The handful of genuinely prompt-based controls (the optional Architect reorder system prompt, AI-disclosure conventions, the LLM narrative-rendering layer) are each paired with a code-side backstop.

The test suite and machine-checked numbers

The whole system is held to a 4,687-test suite, and the documentation numbers are themselves machine-checked: a canonical-facts CI drift gate (scripts/check_canonical_facts.py) fails the build on any backward or forward numeric drift in tracked docs. Recall is regression-tracked at 72/72 (100%) disk and 108/118 (91.5%) memory, on Python 3.12+.

Wazuh integration

For SIEM operations, the server talks to two separate Wazuh services over two distinct auth chains: the Wazuh Indexer (OpenSearch fork, :9200, Basic Auth over HTTPS) holds case/evidence/findings/timeline/IOC/approval indices with shipped templates and ISM policies; the Wazuh Manager API (:55000, JWT) receives the operator-gated IOC egress. Live SIEM mutation (wazuh_publish_iocs) is defended in depth: a single-use scoped egt_ token, provenance-aware false-positive denylists with RFC1918 gating, and the WAZUH_DRY_RUN_ONLY kill switch — an env boolean (default on) checked before token verification, which the model cannot toggle. A batch orchestrator maps findings to alerts (staging indices on dry-run) with a 180-day dedup cache, and ships Findings/Timeline dashboards.

Challenges we ran into

Building an autonomous agent that finds evil without inventing it meant most of our hard problems were honesty problems, not feature problems. Here are the real ones — solved, or honestly bounded.

1. Volatility on raw memory has no "what OS is this?" step

The instinct — ours, and the LLM's — is to start a memory investigation by identifying the image: run something like windows.info / banners, or get_image_info, to confirm the OS and profile. On a raw .mem/.raw/.bin dump that is wrong: get_image_info drives ewfinfo and reads EWF/E01 metadata only, so it returns all-empty on a raw dump, and windows.info/banners are not allowlisted at all. Volatility 3 is profile-less — the kernel symbol table auto-detects on the first windows.* plugin (get_pslist). A populated pslist is the confirmation; an empty pslist plus a kernel.symbol_table_name error is the honest negative ("no profile resolved"). We had to delete the OS-identification step everywhere it had crept into our case guides and teach the agent that the first real plugin call is the profile check. os_id_misuse was one of four recurring bug classes we audited to zero across all case-activation guides.

2. A wrong Volatility plugin id is a demo-breaking bug

run_volatility accepts a plugin name in exactly two forms: a short alias (pslist, malfind, cmdline, netscan, pstree…) or a full canonical id (windows.cmdline.CmdLine). The natural-looking middle form, windows.cmdline, is rejected. We also caught MCP "tools" in early drafts that simply don't exist in our 73-tool surface — get_hashdump/hashdump (we expose no credential-dump capability, so the step is dropped, not faked), get_srum (→ srum_extract), and mmls used as an MCP slot (→ get_partitions/parse_gpt). We resolved this by auditing every documented tool and plugin against the live tools/list rather than against memory, fixing 37 issues across the case guides. The lesson generalized into a standing rule: validate every tool/plugin name against the running MCP, never guess.

3. Heavy Volatility plugins false-timed-out

Plugins like malfind legitimately run long on a large memory image, and they were being killed at the MCP SDK's 180-second default — a false timeout that looked like a tool failure. The fix was to pass the per-tool {timeout: 300000, resetTimeoutOnProgress: true} option so progress resets the clock; a plugin that is still streaming output is not stuck. Bounded and solved, but it cost us a round of confusing "the agent broke" reports before we traced it to the timeout default.

4. Our own false-positive gate refuting our own findings

This is the challenge we are proudest of, because it is the product. Our false-positive gate filters findings on confidence > 0.0 against clean-image baselines, and it repeatedly caught us:

On the VANKO insider-IP-theft case, the FP gate refuted 9 of 19 candidate findings — leaving 10 confirmed. The refuted ones included generic YARA family hits (with_sqlite, XMRIG_Miner) with no PE backing, and a pagefile dropbox.com/s/ string that turned out to be an obfuscator template, not the actor's exfil link. Shipping those as "evidence" would have been hallucinated forensics.
W-168 — two newer detectors emitted Finding records with confidence > 0.0 on skip/error/empty paths, so "ran but had nothing to do" was scored as a confident hallucination on every clean image. Fixed by zeroing confidence on infrastructure-event paths.
W-180 — a YARA bookkeeping/status finding regressed to confidence=0.50 and tripped the clean-image FP gate; fixed to 0.0 with a dedicated regression guard test.
W-270 — a reparse-point junction-name hint fired unconditionally on any My Documents path: correct on Vista+, a false positive on XP-era NTFS where it's a real allocated directory. Fixed to fire only after a successful retry, with a falsifiable acceptance test.

Each of these was a case of the agent almost asserting something it couldn't prove, and the architecture stopping it. We treat refuted hypotheses as honest negatives kept on the record, not as failures swept away.

5. Memory recall is 91.5%, and we refuse to round it up

Our disk per-IOC recall is 72/72 (100%) on the SRL-2018 regression suite — but that figure is partially curve-fit (6 of 7 ground-truth YAMLs were authored from earlier wrapper output), so we disclose it inline and refuse to lead with it. The honest combined number is 108/118 (91.5%) across memory + disk, and the worst band — T1003.002 (SAM credential dumping) at 30/40 = 75% — is exactly where 10 of those 118 IOCs are missed. We also had a humbling failure history: the recall gate sat at 1/7 in mid-April 2026 and climbed milestone-by-milestone to 7/7, and even after "perfect recall" was first claimed, a live re-run dropped a technique back to 6/7 due to plaso non-determinism — which we logged rather than hid. The challenge here wasn't getting a big number; it was building the discipline to publish the small ones.

6. Making the model structurally unable to fabricate

The hardest design challenge was guaranteeing that no fact in a report can originate from a language model — not "the model was told to be careful." The LLM agents (Architect, Critic) only orchestrate; every Finding is authored by a named deterministic MCP wrapper that read bytes off evidence. We had to enforce this in code with controls the agent cannot route around: the read-only Thymus policy that rejects unsafe/out-of-bounds paths before any I/O, a pre/post SHA-256 evidence invariant that cryptographically pins the report to the exact evidence bytes, a deterministic fingerprint halt (a frozenset fixed-point plus a numeric threshold, never a model self-rating) for the Trinity loop, and an HMAC examiner-approval hard stop so a human signs before anything is sealed. The engineering challenge was resisting every shortcut where it would have been easier to let the model summarize — because the moment a fact comes from the model, the courtroom story collapses.

7. Surviving the rendering platform

A more mundane but real one: our diagrams and demo media had to survive the platform. Mermaid renders client-side under strict security levels (no C4, no timeline, no HTML-tag-like tokens in labels), and GitHub is strict too — it rejects foreignObject SVGs, shrinks inline Mermaid, and renders no player for repo-committed MP4s. The fix was a consistent pipeline: render Mermaid to PNG (mmdc) for guaranteed same-origin display, and serve demo video via GitHub Pages with auto-start watch pages rather than relying on inline playback. Tedious, but the alternative was a "broken diagram" first impression for judges.

Accomplishments that we're proud of

We set out to build an agent that finds evil without ever lying about what it found. Here is what we're proud of — every claim below is grounded in the repo, and the honest negatives are part of the point.

Real findings on real staged cases — sealed and examiner-approved. Agentropix ran end-to-end on three fictional-but-realistic DFIR cases: SRL-2018 (multi-host APT IP-theft across the Stark Research Labs estate), SRL-2015 (a SANS APT enterprise case across 8 disk/memory hosts), and VANKO (an insider IP-theft — explicitly not a malware intrusion, which mattered for how we scoped it). These are not toy demos: the SRL-2018 flagship is a live engine run over a real E01 disk image, producing a 166 KB sealed report, a cross-bound 146-entry audit log, and a per-run HMAC session key — with 10 findings APPROVED via an examiner-signed HMAC chain. On SRL-2015 the pipeline aggregated 2,233 findings across 8 host runs, exported 91 IOCs (12 flagged malicious) and a 92-object STIX 2.x bundle, and distilled a 17-finding approved report.

The honest-negatives discipline — the agent refutes its own findings. This is our proudest result. On VANKO the engine raised 19 hypotheses and confirmed only 10 — the other 9 were refuted by its own false-positive gate rather than dressed up as evidence. Our accuracy audit is deliberately unflattering about itself: it labels every component with no measured ground truth as "not benchmarked — no ground truth in repo" instead of inventing a number, flags that the headline cross-modal figure is operator-attested (weaker provenance than the sealed runs), and documents where the engine's control logic has thin isolated test coverage. An agent that volunteers its own limits is the one you can put in front of a courtroom.

Architectural — not prompt-based — guardrails. The integrity guarantees live in code, not in a system prompt the model can talk its way around:

No LLM in the halt path. The Trinity loop's Critic is a deterministic, LLM-free scorer (score = min(1.0, max_conf + 0.25·#correlations), halt at >= 0.85). The model never gets to rate its own confidence.
A read-only evidence boundary (Thymus). Every evidence access passes a policy gate before any byte is read; each decision is logged to a Thymus access trail (146 lines on the SRL-2018 run).
A human-in-the-loop hard stop. Promotion from DRAFT to APPROVED requires a valid nonce + HMAC challenge-response; the approver password is never transmitted, and examiner crypto sign-off sits on a fixed hard-stop deny-list the autonomous agent may never bypass.
Tamper-evident sealing. Reports and audit logs are HMAC-sealed and cross-bound, so a forger can't alter one without breaking the other.

Measured recall we can stand behind. 72/72 (100%) disk-recall regression and 108/118 (91.5%) combined memory recall — and we report the worst band honestly: the T1003.002 SAM band sits at 30/40 (75%). We'd rather show the 75% than hide it.

A complete, governed engine. 73 MCP tools driving 16 real SIFT forensic binaries, a 7-agent swarm over a quorum blackboard, 4,687 collected tests, and Python 3.12+ — all wired into one end-to-end pipeline: acquire → triage → human-approve → seal → report → SIEM (Wazuh/OpenSearch, with a dry-run gate before any live index). That whole chain ran for real, on real images, and the system told us the truth about what it could and couldn't prove.

What we learned

Building an autonomous DFIR agent that a court — not just a demo audience — could trust forced us to unlearn a few things we thought we knew about LLM applications. The hardest lessons all converged on one idea: in forensics, trust has to be a property of the architecture, not a property of the prompt.

Architectural guardrails beat prompt-based ones for forensic trust

Our first instinct, like most teams shipping an agent, was to write better instructions: tell the model to stay in evidence directories, tell it not to approve its own findings, tell it to ground every claim. We learned the hard way that an instruction the model is expected to honor is not a control — it's a hope. A prompt-based guardrail is one the model can reason its way around, and a forensic examiner cannot put "we asked it nicely" on the stand.

So we moved every load-bearing control into code the model cannot reach around. The MCP server is the enforcement boundary — "Thymus policy runs here, not in the agent." The Thymus read-only evidence policy (thymus_policy.py, 52 check_read call sites) means check_write() on evidence unconditionally rejects: the agent literally has no tool to write evidence. Every finding is force-stamped DRAFT by the W-286 gate, which strips any caller-supplied approval.* field, so the model cannot self-approve through any write surface. Bearer auth is fail-closed — _build_app() refuses to start without a token. The one runtime LLM touchpoint that survived — an optional, default-off agent-reorder pass — is wrapped in code that rejects its output unless it exactly equals the deterministic set. The model gets a vote; the code gets the veto. The rule we settled on: pair every prompt-based control with a code-side backstop.

Designing for honest negatives is a feature, not an apology

The most counterintuitive lesson: the moment we stopped trying to make the numbers look perfect, the project got more credible, not less. Our disk recall is 72/72 (100%), and we refuse to lead with it — because we know, and we disclose, that most of that ground truth was authored after looking at wrapper output, so it measures regression stability more than blind detection power. The number we actually lead with is the honest combined one: 108/118 (91.5%), dragged down by a 75% band on SAM credential dumping. On the VANKO insider-IP-theft case, the agent confirmed 10 findings of 19 and let the false-positive gate refute the other 9 — and we kept the refuted hypotheses in the report as honest negatives rather than quietly deleting them.

This only works if the system is built to surface negatives instead of papering over them. We learned to engineer the FP gate as a first-class citizen, and we learned it the painful way: early detectors emitted confident Finding records on clean images (W-168), so "ran but found nothing" was being scored as a confident hallucination. We caught it in our own output, logged it in a public weakness ledger, and fixed it. The discipline that an empty pslist is an honest negative — not a failure to be smoothed over — is now baked into the tools, not asked for in a prompt.

Deterministic agents over the MCP boundary — LLM at the edge, determinism inside

We expected the interesting engineering to be in the agents. It turned out to be in the boundary. The pattern that earned its keep — verified end-to-end — is "LLM at the edge, determinism inside." Every fact the system reports originates from a named, deterministic tool wrapping one of 16 SIFT Workstation binaries, invoked via no-shell create_subprocess_exec, never a shell string. The in-process Trinity-Loop swarm (13 agent classes) is explicitly LLM-free — pure async coroutines with a Critic that halts on a pure function of blackboard state, not on a model's say-so. The LLM does what LLMs are good at (routing a non-expert's plain-language question to the right capability, narrating grounded sections); the deterministic core does what forensics requires (reproducible bytes). Keeping that line sharp is what let us answer the courtroom question — where did this fact come from? — with a file path instead of a shrug.

Human-in-the-loop is a hard-stop, not a suggestion

Autonomy and accountability are in tension, and we resolved it by making one step un-automatable. No DRAFT finding becomes APPROVED without a human examiner completing a real challenge-response: a 60-second single-use nonce plus an HMAC-SHA256 signature derived from the examiner's password via PBKDF2 (600k iterations), gated by a precondition check that the target exists and holds the asserted status. There is no code path that writes an approval without a valid signature — not even for our own demos, where we label every simulated approval "SIMULATED examiner approval (demo only)" precisely because the real control is cryptographic and we won't let a showcase blur that. The lesson: if the human checkpoint can be skipped under deadline pressure, it will be — so we made it impossible to skip rather than impolite to skip.

Provenance grounding is what turns output into evidence

Finally, we learned that "don't hallucinate" is not achievable by asking — it's achievable by making every claim physically traceable to bytes. Wrapper responses carry a raw_stdout_sha256 of the actual subprocess output, so a parsed finding is anchored to the deterministic tool's raw bytes. Reports are sealed with per-run HMAC keys, the audit-log seal is cross-bound into the report before the report seal, and evidence_image_sha256 binds the report to the image bytes — with re-verification CLIs that exit non-zero on any forgery. Anti-hallucination, we concluded, is not a model behavior we coax; it is a chain of custody we compute.

If we had to compress all of it into one sentence: we stopped asking the model to be trustworthy and started building a system where the truthful path is the only path the model can take.

What's next for Agentropix-SIFT

Our roadmap is not a wish list — it is a routed index of commitments already on record as Deferred ADRs (each with a written re-attempt condition), identified replacements in our Design Decisions log, and the weaknesses we name ourselves in the README SWOT. Deferring formally, with an explicit trigger, is itself one of our project rules (ADR-021): we never silently cut scope, and we never put a date on work we have not started. Everything below traces to a source — no vaporware.

Close the memory-forensics recall gap (our active front)

This is the honest negative we are most determined to fix. Disk recall sits at 72/72 (100%); memory recall trails it at 108/118 (91.5%) and is the standing accuracy front. Several detectors that would lift it are already written, wired, and unit-tested but deliberately held back from the live-recall figures so we never report a capability we cannot prove:

Event-window recall (T4 EventID 4624, T5 MFT timestomp). M6.4 opens with an instrumentation sprint, not another design sprint — we land trace.timeline.jsonl_rows_read, priority_hits, and detectors_fired counters to disambiguate the remaining hypotheses behind "the wrapper reports success, the agent reports zero findings" (ADR-W051/W054-defer, ADR-M6.3-residual-gap). One hypothesis (a silent psort flag failure) is already confirmed and fixed.
Volatility-backed process-injection detection (T6) and a YARA agent for Cobalt Strike stager artifacts (T2). Both stay honest MISSes until the detector fires — we refused to soften the ground truth to manufacture a hit, because "changing the GT would misrepresent the evidence" (ADR-W052-T6/T2-defer).
A Plaso output-schema contract test to pin timestomp sentinel tokens against upstream drift, so a silent regression can never pass unnoticed (ADR-W054-defer, residual risk).

Restore in-memory credential triage

The W-072 credential wrapper (SAM/LSA/MSCache) was lost to a Volatility version pin. The accepted path forward keeps the upstream pin and extracts credentials post-M7 via impacket-secretsdump LOCAL against registry hives carved offline (ADR-014). It was deferred for a defensible reason: our submission was recall-driven, not credential-driven.

Harden Active Response with a two-person rule

Today a single examiner-confirmation gate (ADR-019) is the only safety boundary on any active-response action. When the first AR endpoint is designed, a follow-up ADR must either ratify a dual-operator co-sign for destructive=true endpoints or document why single-confirmation suffices — before any AR endpoint becomes callable from the MCP wrapper layer (ADR-021). This keeps the human-in-the-loop guarantee architectural, not aspirational.

Promote the LLM Task Router and per-call policy resolution

Two designs exist on paper with no committed sprint, listed so readers don't mistake documentation for shipped code: an LLM Task Router that classifies task structure (parallel / sequential / hybrid) without letting the model execute the task (ADR-009, Proposed — NOT shipped), and a contextvar-based per-call Thymus policy to replace today's module-global evidence-gate policy (Design Decisions §3).

Broaden case and SIEM coverage

We ship validated runs on a small set of real cases (SRL-2018 APT IP-theft, SRL-2015, VANKO insider IP-theft). Extending the case-activation library — and the Wazuh/SIEM IOC-egress path that already pushes confirmed findings as additive CDB unions — is a natural next front; each new case both proves and stresses the recall gates above.

Packaging, reporting UX, and the demo cut

The multi-tier report engine (exec / business / technical) already shipped (report_export, ADR-024 Phase 5), so it is not future work — the remaining SWOT gap is broader commercial case-file UX (we are a triage engine, not a case-management product like AXIOM or CADO), for which no ADR yet commits a plan. The installable agentropix-mcp package and GitHub release line continue to be polished. The one outstanding hackathon deliverable is the ≤5-minute narrated demo: our full 8:27 evidence walkthrough exceeds the cap, so a sub-5-minute narrated cut showing one self-correction (an agent reporting an honest negative) is the immediate open item.

What is deliberately not on this roadmap

So absence reads as a decision, not an oversight: no host-collection / agent-fleet ecosystem (we are a read-only post-collection triage consumer, by design — an honest scope boundary, not a gap we are promising to fill), and no dates or version promises — every item above fires on a condition (a sprint kickoff, an instrumentation result, a detector landing), never a calendar.

--- README LINK SNIPPET (apply after the Devpost page is published) ---

Drop this line into README.md immediately below the Featured video / Vimeo link and above (or beside) the top status badges, so judges arriving from Devpost land back on the story:

> 📝 **Devpost story:** [Agentropix-SIFT on Find Evil!](DEVPOST_URL_HERE)

Note: replace DEVPOST_URL_HERE with the published https://devpost.com/software/... URL once the project page is live. Place it in the same top block as the existing Final_Video/SRL-2015-EVIDENCE-presentation.mp4 / Vimeo (https://vimeo.com/1201031111) feature so the README's "watch the demo" and "read the story" calls-to-action sit together.

--- DEVPOST GALLERY / MEDIA SUGGESTIONS (paths verified to exist) ---

Upload these existing repo artifacts to the Devpost gallery; lead with the architecture diagram and the VANKO honest-negatives material, since those carry the project's identity:

Architecture diagram (PNG) — docs/02-architecture/assets/architecture-diagram/architecture-diagram.png (625 KB, the judge-verified main diagram). Shows the "LLM at the edge, determinism inside" boundary at a glance — use as the primary gallery image.
SRL-2018 case-report diagrams (PNG) — docs/12-CASES-REPORTS/srl-2018-report/diagrams/d1.png … d5.png. Real APT IP-theft triage results with a visual attack narrative.
VANKO case-report diagrams (PNG) — docs/12-CASES-REPORTS/vanko-report/diagrams/d1.png … d5.png. Pairs with the honest-negatives story (10 confirmed of 19, 9 refuted).
Wazuh dashboard captures (PNG) — docs/12-CASES-REPORTS/vanko-report/wazuh/02-findings-approved-10.png, 06-mitre-attack.png, 07-threat-hunting.png. Show approved findings/IOCs landing in the SIEM.
Demo video — link the featured evidence walkthrough on Vimeo (https://vimeo.com/1201031111) in the Devpost "Try it out" / video field; use a frame from it as the video tile.

Built With

.net
anthropic-claude
asyncio
bash
bmad
c
claude
csv
defusedxml
evtx
exiftool
extract-msg
ez-tools
fastmcp
gemini
github
hashdeep
http
httpx
json
json-lines
json-rpc-2.0
libpff
mitre-att&ck
model-context-protocol
oletools
openclaw
perl
plaso
pydantic
pytest
python-3.12
regripper
rich
rust
sans-sift-workstation
server-sent-events
sqlite
tailscale
tenacity
typer
ubuntu-linux
volatility3
wazuh-api
yara