CoverageForge

CoverageForge maps your Splunk data against MITRE ATT&CK, identifies detection gaps, and auto-generates deployment-ready SPL rules - turning days of manual detection engineering into minutes.

Comment

The Problem

Every security team using Splunk faces the same challenge: they have data, but they don't know what attacks that data can actually detect. Mapping sourcetypes against MITRE ATT&CK, writing SPL for each gap, and deploying it to Splunk is a process that takes skilled detection engineers days - and most organizations have dozens of data sources they have never fully analyzed.

The typical workflow looks like this:

  • Run fieldsummary manually against each sourcetype to understand available fields
  • Cross-reference fields against MITRE ATT&CK to figure out which techniques could be detected
  • Write SPL detection searches from scratch for each identified gap
  • Format each search as a savedsearches.conf stanza with MITRE annotations
  • Deploy to Splunk, configure severity, set cron schedules

This takes a skilled detection engineer 2-8 hours per data source. CoverageForge automates the entire lifecycle end-to-end in under 5 minutes.

What CoverageForge Does

CoverageForge is a 7-phase agentic AI pipeline that connects to your Splunk instance and handles everything automatically.

Phase 1 - Data Inventory Uses tstats via the Splunk MCP Server to discover every indexed sourcetype and event count across all indexes.

Phase 2 - Detection Inventory Pulls all existing saved searches via MCP and extracts MITRE ATT&CK coverage from the official action.correlationsearch.annotations field used by Splunk Enterprise Security.

Phase 3 - Gap Analysis Cross-references existing coverage against the MITRE ATT&CK Enterprise matrix (12 tactics, 84 techniques) to produce a baseline coverage percentage and gap list per tactic.

Phase 4 - Field Discovery Runs fieldsummary per sourcetype via MCP with adaptive head limits to stay within the 60-second MCP timeout on large datasets.

Phase 5 - MITRE Classification Uses Foundation-Sec-8B (Cisco/Splunk's security-tuned AI model) to classify which attack techniques are detectable from available fields and sample values.

Phase 6 - SPL Generation Calls Splunk AI Assistant (saia_generate_spl) via Splunk Cloud MCP for each use case. Claude Sonnet 4.6 handles descriptions, conf stanzas, and any use cases SAIA could not generate. Each use case is tagged with spl_source showing whether SAIA or Claude generated it.

Phase 6b - Validation Runs every generated SPL against real Splunk data via MCP to confirm it executes without errors and returns results.

Phase 7 - Output Generates an interactive HTML coverage report with Chart.js visualizations, a deployment-ready savedsearches.conf, structured use_cases.json, and a field inventory CSV.

Deploy Uses the Splunk Python SDK (splunklib.client) to create a coverageforge_detections app and deploy all generated searches directly into Splunk with one command.

Splunk AI Capabilities Used

Capability How Used
Splunk MCP Server All data collection at runtime - splunk_run_query, splunk_get_knowledge_objects, splunk_get_metadata
Foundation-Sec-8B MITRE ATT&CK technique classification from sourcetype field data
Splunk AI Assistant (SAIA) saia_generate_spl MCP tool for SPL generation via Splunk Cloud
Splunk Python SDK One-click deployment via splunklib.client with two-step app creation pattern

Challenges

MCP timeout on large sourcetypes The MCP Server has a 60-second hard timeout. Large sourcetypes with millions of events would exceed this during fieldsummary. Solved with adaptive head limits based on event count tiers.

SAIA response format SAIA requires prompt as the argument key (not query), and returns SPL inside a nested structuredContent.results[0].response structure rather than as plain text. Both required significant debugging since neither is documented clearly.

Claude JSON truncation For large batches, Claude sometimes hits the token limit mid-JSON response. Built a bracket-depth parser that recovers all complete objects from a partial response so no results are silently lost.

MCP token audience claim The Splunk MCP token must be generated from inside the MCP Server app, not from Settings > Tokens. The two token types use different audience claims and produce different error messages - the root cause took hours to identify.

splunk-sdk 3.0.0 breaking change Creating saved searches scoped to a specific app requires a two-step connect pattern: connect without app scope to create the app, then reconnect scoped to the app for saved search operations. Single-step connect causes SavedSearches.__init__() errors.

What I Learned

Detection engineering is fundamentally a data coverage problem, and AI is well-suited to solve it systematically. Foundation-Sec-8B's security domain training produces significantly better MITRE technique classifications than a general-purpose model given the same field data. SAIA generates production-quality SPL that accounts for field aliasing and edge cases that generic prompting misses. The combination of MCP for data access, Foundation-Sec for classification, SAIA for SPL, and the Python SDK for deployment creates a complete agentic loop that requires no manual steps.

Built With

Share this project:

Updates