AlertFatigue

Stop chasing ghosts. AlertFatigue uses AI to tell your SOC which alerts are real threats and which are just noise, then writes the suppression rules for you.

Comment

Inspiration

Every SOC analyst I've spoken to describes the same Monday morning: 847 alerts waiting, coffee going cold, and the sinking feeling that the real threat is buried somewhere in the noise. Security teams don't lose to attackers because they lack data. They lose because they're drowning in it.

The average SOC spends 25-50% of its time investigating false positives. That's not a detection problem, that's a signal-to-noise problem. And yet every existing tool either requires expensive ML infrastructure to solve it, or pushes analysts to manually tune suppression rules one by one based on gut feel.

I wanted to build something that used Splunk's own data and Splunk's own AI to close that loop automatically.

What it does

AlertFatigue connects to your Splunk instance, pulls 30 days of triggered alert history via the Splunk MCP Server, and runs an agentic classification pipeline on every alert pattern it finds.

For each alert group it identifies:

  • Classification: TRUE_POSITIVE, FALSE_POSITIVE, or REVIEW_NEEDED
  • Confidence score with plain-English reasoning
  • Auto-generated SPL suppression clause for noise alerts
  • Estimated volume reduction if the suppression rules were applied

The result is a suppression report your team can review in 10 minutes and push back to Splunk saved searches with one click, turning days of manual tuning into a single automated workflow.

How we built it

The stack is intentionally lean:

Backend (Node.js/Express)

  • Splunk MCP Server (run_splunk_query) for real-time alert pulls
  • Splunk Python SDK for bulk 30-day historical analysis beyond MCP's event limits
  • Foundation-Sec-1.1-8B-Instruct (Splunk hosted model) for security-aware classification and suppression SPL generation
  • gpt-oss-120b for complex multi-condition SPL generation where instruction following precision matters

Frontend (React + Tailwind)

  • Alert group table with color-coded classifications
  • Generated suppression rules in syntax-highlighted code blocks
  • One-click save back to Splunk via MCP
  • Summary banner showing projected alert volume reduction

The core agentic loop: pull alerts → aggregate by pattern → classify with Foundation-Sec → generate suppression SPL → present for human review → optionally write back to Splunk. Every step uses Splunk-native infrastructure with no external AI dependencies.

Challenges we ran into

The 1000-event hard cap on MCP's run_splunk_query was the first wall we hit. Thirty days of alert history in any active environment blows past that immediately. We solved it by using aggregation-first SPL (stats count by alert_name) through MCP for pattern detection, then falling back to the Python SDK for raw event access when needed.

Foundation-Sec JSON consistency was the second. The model returned valid JSON roughly 70% of the time unprompted. The remaining 30% had markdown fences, truncated output, or missing fields. We built a multi-pass fallback parser: strip fences, attempt parse, and if that fails, re-prompt with an explicit schema reminder. This brought reliability to ~97%.

Developer License wait time cost us nearly half a day at the start. The approval process has no status indicator and the confirmation email is spam-prone. We eventually got through it but it's a real barrier for hackathon participants on a deadline.

Accomplishments that we're proud of

The suppression rule generation actually works on real alert data. It's easy to demo a classification UI. It's harder to produce syntactically valid, environment-specific SPL NOT clauses that a real SOC analyst would trust and push to production. Getting Foundation-Sec to generate suppressions that preserved the original search logic while excluding the noise pattern felt like the moment the project became real.

We're also proud of keeping the entire pipeline on-platform. No data leaves Splunk infrastructure. For a security product, that's not a nice-to-have, it's a requirement.

What we learned

Foundation-Sec-1.1-8B-Instruct is meaningfully better than general-purpose models at security classification tasks. It correctly identified CI/CD service account noise, scanner traffic patterns, and scheduled task signatures that gpt-oss-120b flagged as suspicious. Domain tuning at the model level is real and measurable, not just marketing.

We also learned that the hardest part of agentic security tools isn't the AI, it's the write-back. Reading from Splunk is well-supported. Writing back (updating saved searches, creating suppression rules) requires working around MCP's current read-only toolset via the REST API directly. That gap is the single biggest thing holding back a new generation of Splunk automation agents.

What's next for AlertFatigue

  • Scheduled runs with Slack/email digest: run every Monday at 8AM, send the suppression report before the analyst's first coffee
  • Feedback loop: analysts mark suppressions as accepted or rejected, fine-tuning future classifications for their specific environment
  • MITRE ATT&CK mapping: before suppressing any alert, cross-reference the pattern against ATT&CK techniques to ensure we're not silencing a real TTP
  • Multi-instance support: enterprise SOCs run multiple Splunk deployments; AlertFatigue should aggregate across all of them

Built With

Share this project:

Updates

Submission history