Inspiration

When a new attack is reported, a defender has to turn the write-up into a detection rule. AI can now produce a Sigma rule in seconds, and that is exactly the problem: a rule that looks valid can still miss the attack, or fire on normal activity, and a generated rule almost always arrives with no evidence either way. A confident-looking YAML file is not a detection.

We wanted the opposite of "here is your rule". We wanted "here is your rule, here is what it did when we ran it, and here is where it failed."

What it does

Shakedown turns a threat report into a Sigma rule and then tests it instead of handing it over unchecked.

Tavily reads the report at run time and searches for corroborating sources. Web text is labelled untrusted before the model sees it.
NVIDIA Nemotron 3 Super 120B on Nebius Token Factory extracts a typed spec. Every observable must come with a quote that appears word for word in the report, or it is dropped. The model then writes the rule as JSON, and our code renders the Sigma YAML.
sigma check validates it.
A Nebius Token Factory Sandbox (a disposable microVM with networking off) runs SigmaHQ's own checker against a labelled attack log and against 481 benign Windows log files: 129 development files the loop sees and 352 held-out files the model never sees.
If the rule missed the attack or matched benign events, the exact evidence goes back to the model for a bounded repair, at most three.

The page shows every attempt, the diff between attempts, the results with their denominators, and SigmaHQ's expert rule for the same behaviour as a baseline. A rule that never passes is shown as not passing.

A rule gets three separate answers, never one pass: rule validity, attack validation, and environment preflight. The third one is new. The Integrations workspace lets an analyst connect a SIEM they already use (Splunk, Elastic, Microsoft Sentinel), prove access with a read-only account, see the exact native query before anything runs, and run a bounded, read-only search of their own history. It recommends and deploys nothing.

How we built it

Python and FastAPI for the backend, vanilla JavaScript for the interface (no build step), pySigma for converting rules to SPL, Lucene and KQL.
Acceptance is deterministic code, not a model opinion: the attack file must be detected and the development benign logs must produce zero matches. Rule ids, scoring, the spend caps and the rendering of YAML are all plain code. The model's output is parsed as data and never executed.
Thirty reference rules produced identical match counts in the Sandbox and in a local container, which is what let us trust the Sandbox as the evaluator.
The Integrations workspace has provider adapters behind a registry, a destination policy against SSRF and DNS rebinding, a secret store, a read-only query guard per language, and a local API guard. Microsoft sign-in uses OAuth with PKCE as a public client, so no client secret exists anywhere.
Every claim in the app maps to an inspectable record: each run keeps its rule text, hashes, validation output, per-file results, sources and token counts. Over 490 tests run by default, and 22 opt-in live tests run against real Splunk and Elastic containers.

Challenges we ran into

The model returned empty answers. With structured output on, Nemotron sometimes returned a valid-looking but empty JSON skeleton. We measured it, then added resampling and a last-resort fallback call. Without that, "the model wrote nothing" would have looked like "the rule failed".
Web search leaked the answer. Searching for a SigmaHQ-derived scenario can return copies of the very rule being written. We found this ourselves, showed that one early recording had copied its reference rule, and added source screening, a guard that refuses model requests carrying the target rule id, and neutral labels in repair feedback. The affected recordings are kept, labelled, and not used as evidence. The frozen benchmark never used web research and was not affected.
Honest evaluation is harder than good-looking evaluation. After a repair the model has seen the missed events, so that detection is not independent. We froze twenty new tasks before running any model on them and report first attempts separately from repaired ones.
Verifying integrations without overclaiming. We could test Splunk and Elastic against throwaway local instances, but not a production SIEM or a real Azure tenant. Mock tests are labelled as mocks and never counted as verification.
A security review of our own sign-in code found two real defects (a non-ASCII state value caused a server error, and the callback did not check its own address). Both are fixed and covered by tests.

Accomplishments that we're proud of

A result we can stand behind, and one that is not flattering. On 20 SigmaHQ tasks frozen before any run, first attempts detected the attack in 6 to 7 of 20 (7 of 19 completed in one run, 6 of 20 in another), and after up to three repairs in 14 of 20 in both. SigmaHQ's expert rules detected 20 of 20. Shakedown does not beat them. What it adds is a tested rule with evidence, produced from a description.
The product shows failure as clearly as success. The video includes a run that did not pass in four attempts.
Every status is stated exactly: Splunk Enterprise and Elastic are live-verified only on throwaway local test instances with simulated events. Splunk Cloud is implemented but not live-verified. Microsoft Sentinel is mock-tested, not live-verified.
We audited ourselves publicly and kept the evidence of what went wrong.

What we learned

Valid is not detecting, and detecting a lab sample is not detecting in production. Keeping those as separate verdicts changed how we designed everything else.
Evidence is the product. A rule with a table of denominators is more useful to an analyst than a better-looking rule without one.
Treat model output and web text as untrusted data, and treat your own evaluation setup as something that can leak.
A mock test proves your code handles the shapes you expected. It says nothing about the real service.
Structured output is not a guarantee of content; measure what the model actually returns.

What's next for Shakedown: Sigma detection rules with evidence

Verify the Sentinel and Splunk Cloud adapters against real tenants, and Splunk and Elastic against a real Sysmon or Winlogbeat mapping, before claiming more than we have shown.
More than one attack sample per rule, and benign logs from more than a few lab machines.
A hosted multi-user mode, which needs real accounts, tenant isolation and a secret vault. It is intentionally disabled today because those do not exist yet.
More log sources beyond process creation, and a QRadar adapter.

Code and documentation: https://github.com/Unknown1502/Shakedown