SIFTArena

SIFTArena - Bring your own: attack chain, workflow, and model. Benchmark LLM agents on forensic reconstruction against ground truth. Reproducible playbooks. Quantified accuracy.

Comment

Inspiration

I wanted to build a benchmarking tool for research purposes because I couldn't find any existing projects.

What it does

Bring Your Own

  • Attack Chain — define custom adversary emulation plans with granular TTP sequencing.
  • Workflow — supply your own forensic analysis pipeline for evaluating agent performance.
  • Models — plug in any LLM (OpenCode, Claude, GPT, etc.) for agentic analysis.

Attack Chain Engine

  • Highly configurable attack chain with reliable ground truth.
  • Each run produces deterministic artifacts tied to known TTPs for reproducible benchmarking.

Playbooks

  • Importable / exportable playbooks designed for reproducibility.
  • Share, version, and reuse attack-and-analysis configurations across teams.

Workflow Benchmarking

  • Configurable workflows for benchmarking effectiveness in reconstructing timelines of events from artifacts.
  • Measure agent accuracy against ground truth across different models and attack scenarios.

How we built it

Used ludus range for vm management, used caldera for administering atomic red team abilities, and used opencode for execution harness.

Challenges we ran into

  • Disabling Windows Defender was difficult. Spent days trying to figure it out and ended up adding a domain controller to persistently disable it.
  • NTP sync is important after reverting to a previous proxmox snapshot or else, the downstream artifact analysis will use stale timestamp ranges and not identify any artifacts.
  • I had contamination issues of my ground truth somehow ending up inside the agent session. Needed to have clean separation between ground truth and agent's own analysis from the artifact data.

Accomplishments that we're proud of

Finishing on time.

What we learned

I learned that you need to enable logging on the target machine for the artifact to even show itself in the first place.

What's next for SIFT arena

entity grounding and normalization, implementing knowledge graphs for enriched agentic inputs, figuring out a benchmark scoring system for comparison visuals.

Built With

Share this project:

Updates