-
-
Hexbreaker architecture: the 5-role Court, deterministic no-LLM Judge, and six in-code safeguards.
Inspiration
What it does
How we built it
Challenges we ran into
Accomplishments that we're proud of
What we learned
What's next for Hexbreaker
Hexbreaker — two products in one repo. Hexbreaker Forge is the Accuracy Benchmarking Framework the Find Evil! brief asks for (starter idea #5), and Hexbreaker Court is a 5-role adversarial DFIR agent built on top, graded by Forge like any other agent.
Inspiration — starter idea #5
The brief is blunt about why this hackathon exists: "Protocol SIFT works. It also hallucinates more than we'd like." And the organizers name the missing piece outright — idea #5, the Accuracy Benchmarking Framework: "Create a test harness that runs against known-good data with documented ground truth, then scores accuracy, false positive rates, and hallucination frequency. The community needs this benchmark to measure progress."
Forge is that framework. We built it because the accuracy problem is structural: the
only visible competitor with a published 100% F1 (dhyabi2/findevil) claims it on the
same one dataset every visible DFIR-agent entry we reviewed leans on — NIST's CFReDS
"Mr. Evil" Hacking Case. We independently re-measured their pipeline under hackathon
constraints (DeepSeek on Ubuntu instead of Gemma + SIFT-VM) and got 0.0% F1 — their
100% holds only on their original (model, OS, scaffolding) tuple. That's dataset
saturation: when a community converges on one corpus, agents converge on its idiosyncrasies
and "F1 = 100%" stops meaning what people think. You can't measure progress on a benchmark
everyone has overfit.
What it does
Forge synthesizes Windows DFIR cases from a 32-bit seed: seed → manifest.json +
answer_key.json + mock_outputs/, deterministic and byte-identical across hosts. Six
artifact templates ship (timestomp, registry_persistence, multi_artifact, browser,
prefetch, amcache), each with genuine 2-tool corroboration and an answer key withheld from
the agent. Anyone can hexbreaker generate --seed N, point any DFIR agent at the case, and
score it — false positives, planted-bait confirmations (the hallucination metric), and
missed findings all fall out of one exact (artifact_kind, target) tuple match.
Court investigates synthetic and real evidence (Forge cases and the genuine NIST
.E01) and emits {artifact_kind, target, verdict ∈ {CONFIRMED, CONTESTED, REJECTED}},
every finding traceable to the exact tool execution that produced it. Five roles cooperate
and adversarially check each other: Prosecutor (files a Claim), Defender (demands
corroboration, files a Verdict), Witness (independent toolset, called on CONTESTED), a
deterministic Python Judge with NO LLM (rules JR-01..N gate every accept/reject), and
a Provocateur (one prompt-injection payload per case; Judge JR-02 checks every round).
Honesty machinery — six safeguards, all in code, none in prompt
Step-ID referential integrity, a forced tool-call FSM, a strict Pydantic schema
(extra="forbid"), a SHA-256 hash chain, HMAC-SHA256 signing (PBKDF2 600K, MIT pattern
ported from AppliedIR/Valhuntir with attribution), and the runtime Provocateur. Each has a
paired bypass test that attempts the attack and asserts it fails. Self-correction is
architectural: the corroboration rule lives in the Judge, so a CONFIRMED verdict citing a
single tool kind is downgraded to CONTESTED in code — replayable via
scripts/demo_self_correction.py. The model may think anything; it may only cite what
the orchestrator can prove exists.
Measured results (honest flagship)
Two HMAC-signed Forge sweeps (40 runs/mode): F1 ≈ 0.95–0.975 normal / ≈ 0.475–0.525 under
maximum adversarial pressure (planted artifacts + runtime prompt injection every round).
0 planted artifacts confirmed across all 80 provocateur runs; 160/160 transcripts
chain+HMAC verified. On the real NIST .E01, the multi-round Court recovered all 4
deleted recycle-bin executables via fls+INFO2 cross-corroboration — 4/4, P/R/F1 = 1.0,
fp_planted = 0, across 5/5 signed runs. Honest scope: that's ~1 of the case's ~31
question-families; since NIST has no planted decoys, that 1.0 is recovery accuracy, not
bait-resistance.
Withdrawn number. An earlier batched run reported 95.08% F1. It is withdrawn — that pipeline injected literal ground-truth answers into the prompt, so it measured string-copying, not forensics. The injection is removed; we cite it only as this caution.
How we built it
Claude Opus 4.8 (build collaborator) + DeepSeek (the live Court's reasoning model, OpenAI-compatible, 429-aware retry); Pydantic schemas; hash-chained JSONL transcripts; a deterministic Judge. Find Evil! starter approach #3 (Multi-Agent) with the load-bearing twist that the Judge is deterministic Python. Evidence is read-only by construction; every security boundary is architectural, not prompt-based, and each is bypass-tested.
What's next
Wider real-disk NIST coverage; N=50 sweeps; a multi-template leaderboard; an adversarial self-hunting harness that attacks Forge's own cases to keep them un-gameable; opening Forge to the community.
Built by Alexander Sorrell (@Alexander-Sorrell-IT) with Claude Opus 4.8. MIT-licensed.
Log in or sign up for Devpost to join the conversation.