Rulesmith

Every SOC turns vendor threat reports into detection rules by hand: read the report, work out what is observable, write the logic,guess the field names,and ship it untested. RuleSmith does it for you.

Comment

Inspiration

Every SOC turns vendor threat reports into SIEM detections by hand: read the report, work out which behaviours are actually observable, write the logic, guess the field names, write tests nobody has time for, and hope it is not too noisy. It takes hours per report and the result is rarely tested. Microsoft shipped a benchmark for exactly this task in March 2026 (CTI-REALM); the best frontier models score around 0.64, and the published systems (LLMCloudHunter, AutoSigma, Sigma-Hunter) all stop at "an LLM judge says it looks fine". We wanted an agent whose rules are proven by execution, not reviewed by another model.

What it does

Give RuleSmith a report URL, pasted text, or a query like "latest Qilin ransomware TTPs". It:

  1. fetches the intel with Tavily (Search to discover recent reporting, Extract to turn advisories and vendor blogs into clean text),
  2. extracts attacker behaviours with evidence quotes, ATT&CK ids and IOCs (Nemotron 3.5 Lightning, strict JSON schema, thinking off),
  3. plans which behaviours deserve rules, on which logsource, keyed on which fields, and which to skip and why (Nemotron 3 Ultra, thinking on),
  4. writes each Sigma rule and its positive and negative test events (Nemotron 3 Super, grounded on SigmaHQ rules retrieved for the same logsource and technique),
  5. validates with no LLM: pySigma parse, a field dictionary for ~110 logsources, the ATT&CK index, compile to SQLite, execute against the tests and a 2,390-event benign baseline that deliberately contains SCCM running encoded PowerShell, installers writing Run keys, admins running whoami and schtasks, and rundll32 loading real control-panel DLLs,
  6. feeds failures back to Super with the exact event it got wrong, per selection and per field,
  7. red-teams the rule with Ultra-generated evasions (flag aliases, casing, SysWOW64 paths, quoting tricks) and hardens it again,
  8. ships a deployable pack: Sigma, Splunk SPL, Elastic Lucene, Microsoft KQL, tests.jsonl, an ATT&CK Navigator layer and an evidence-linked report.

A rule is validated only if it parses, compiles, catches every attack event including the red-team variants, stays silent on every benign look-alike, and has zero hits on the benign baseline. Anything short of that ships as needs review with the exact failing tests and the reason attached, so the fix is a minute's work, not an afternoon's.

The web UI streams every stage live, shows which Nemotron tier did what with tokens and cost, and gives every rule a card with its ATT&CK links, pass/fail counts, copy and download buttons for each SIEM dialect, the tests with a caught/blocked verdict per event, the refine history, the evidence quotes, and the evasions the red team judged beyond the logsource's reach, so the blind spots are on the page rather than hidden.

How we built it

Python 3.12, FastAPI with Server-Sent Events, and a no-framework front end that also runs as a static page (the hosted demo plays recorded runs with no key). All inference goes through Nebius Token Factory's OpenAI-compatible endpoint with response_format: json_schema and enable_thinking toggled per stage; model ids are resolved at start-up against /v1/models so renames cannot break the app. Validation is deterministic: pySigma's SQLite backend (the engine behind Zircolite) turns a rule into SQL that runs against an in-memory table of events, so passing here means the rule behaves the same way in production.

Tiered routing is the product: Lightning for extraction (1M context, cheapest, no trace needed), Super for the write-and-refine loop (runs most often, strong JSON adherence), Ultra for the two places where judgement matters (what to detect, and how an attacker would evade it). A five-rule run costs about $0.15.

82 tests run without network against a scripted fake model, including a too-broad rule that must be refined using benign-corpus evidence; a headless-browser smoke test drives the UI through a live run and a replay with zero console errors.

Challenges we ran into

  • Making the benign corpus adversarial enough to matter. A naive -enc rule must fail, because SCCM and Intune legitimately run encoded PowerShell. The corpus was generated specifically to contain the things that fool first-draft rules.
  • The model's own tests disagreeing with its rule. The main cause of needs-review rules is a first-draft test that describes a parent process the rule never keys on, or an encoded command the rule expects in clear. We let the refine stage retire a red-team variant it proves unreachable for the logsource, or correct a test event that is itself wrong, but capped, guarded (the author's tests can never be retired, red-team tests can never be edited, a corrected positive may only add fields its siblings already carry), and always kept on the rule with the reason, so nothing is quietly dropped.
  • Structured output in the wild. Model ids differ in casing across Nebius pages, so we resolve them at runtime. The client degrades from json_schema to json_object to prompt-only and repairs invalid JSON with one extra call. Stray quotes in YAML keys and missing condition lines get a deterministic repair pass before the model is asked again.
  • Non-determinism. The same refine evidence produced 7/7 and 5/8 on consecutive calls of Super, so we sample refinement colder, keep the best draft, and never let a round regress.
  • No Nemotron fine-tuning on Token Factory (checked: Llama, Qwen, DeepSeek and gpt-oss only), so instead of distillation we ground the author and refine prompts on SigmaHQ rules retrieved per logsource and ATT&CK technique. That lifted first-try passes from 8 to between 9 and 13 of 25.

Accomplishments that we're proud of

Across the five recorded reports: 25 rules, 15 validated under the strict definition, 187 red-team variants, zero false positives on the 2,390-event benign baseline, $0.73 and seven minutes for the whole set. A real CISA advisory (AA24-241A, fetched through Tavily) came out at 3 of 5 validated with zero baseline hits for $0.13; the demo video is that run, live. Every rule converts to four SIEM dialects with the evidence sentence from the report attached.

What we learned

Execution-based validation catches what LLM judges miss: rules that look right and fire on SCCM, tests that look right and contradict the rule. Tiered routing is a product feature, not a cost trick: Ultra for judgement, Super for the loop, Lightning for extraction, and showing the split builds trust. And honesty about the 40% that does not clear the bar is worth more than hiding it; a needs-review rule with its failing test attached is still an hour saved.

What's next

Pull-request mode against a SigmaHQ-style repository, CTI-REALM scoring so the accuracy claim is comparable, distillation of Super-written rules into a Lightning fine-tune once Nemotron bases are fine-tunable on Token Factory, and Token Factory Sandboxes for running the validator in isolation.

Built With

Share this project:

Updates

Submission history