BattleOps

Agentic security review on every PR — Splunk feeds in historical context before the verdict and captures every finding after, allowing developers and security team to keep common references per issue.

Comment

Inspiration

After COVID-19, something shifted in the global threat landscape that never shifted back.

Lockdowns forced a generation of businesses online overnight, like small manufacturers in Lagos, legal firms in Johannesburg, logistics companies in Warsaw, family-owned retailers across the American Midwest. Most had no security team. Many had no IT budget. All of them suddenly had an internet-facing footprint. And attackers took notice.

The numbers tell a story. The FBI's Internet Crime Complaint Center recorded over $10.3 billion in cybercrime losses in 2022 alone, a record at the time. Ransomware gangs pivoted from targeting Fortune 500 companies (who fight back hard) to SMEs and public institutions, who often just pay for it. In Nigeria, EFCC systems and multiple state government ministries were breached in successive years; fintech platforms serving millions of unbanked users were targeted precisely because they moved fast and reviewed code slowly. In South Africa, the 2021 Transnet ransomware attack shut down container terminals at the Port of Durban for weeks. The South African Department of Justice was hit the same year and courts couldn't process cases. In the US, Colonial Pipeline (2021) triggered fuel panic across the Eastern Seaboard from a single compromised credential. Change Healthcare (2024) took down prescription processing for a third of American pharmacies for weeks. In Europe, the Irish Health Service Executive was crippled by Conti ransomware, and cancer screenings halted, patient records locked for months. Royal Mail UK (2023) couldn't process international shipments for over a month. In China, ICBC's US broker-dealer arm (2023) was hit by LockBit and temporarily couldn't settle US Treasury bond trades, a reminder that even financial infrastructure isn't immune. The MOVEit vulnerability (2023) alone compromised data at over 1,000 organizations globally like BBC, British Airways, Shell, dozens of US federal agencies, not from a targeted attack, but from a single unpatched file-transfer tool.

But that era of attacks, as catastrophic as it was, was still roughly symmetric. A human attacker found a vulnerability, wrote an exploit, ran a campaign at human speed.

The AI era broke that symmetry entirely.

AI agents can now enumerate attack surfaces, write exploit code, test payloads, and orchestrate multi-stage intrusions at machine speed with near-zero marginal cost per target. Black-hat operators are already using LLM pipelines to generate phishing campaigns personalized at scale, fuzz APIs automatically, and adapt attack chains in real time based on defensive responses. Meanwhile, on the defender side, the same AI tools accelerating development, like Copilot, Cursor, Claude, are producing code at a rate that vastly outpaces human review. Studies consistently show LLM-generated code introduces security vulnerabilities at a measurable rate. A startup can now ship a product that's 70% AI-generated in weeks. The attack surface grows faster than any human team can audit it.

The asymmetry is stark: attackers get AI-scale offense for free. Defenders still largely rely on human-speed code review.

BattleOps was built to close that gap, with automated security review on every PR, with findings routed into the same operational platform that security teams already use to detect and respond to threats. If AI is being weaponized to attack at scale, the only credible answer is to use AI to defend at scale too.

What it does

BattleOps automatically reviews every pull request for security vulnerabilities and routes findings directly into Splunk, turning your SIEM into the operational brain for code-level threat intelligence.

When a developer opens a PR, BattleOps:

  1. Fetches the diff via GitHub App webhook
  2. Scans for secrets with gitleaks (150+ detection rules) and checks all changed dependencies against the OSV vulnerability database for known CVEs
  3. Queries Splunk via MCP: Before the AI agent forms an opinion, it calls the Splunk MCP Server to ask: have these files had prior confirmed findings? Do any CVEs in this diff match active alerts in the environment? Historical operational data is baked into the reasoning and planning from the start, not bolted on afterward
  4. Runs the AI analysis agent (DeepSeek): a self-directed tool-calling agent that reads full source files, traces taint flows across the diff, and uses MITRE-mapped attack methodologies to generate findings grounded in both the code and Splunk's memory
  5. Verifies every finding: an adversarial verifier agent challenges every HIGH/CRITICAL result, arguing against the primary agent's conclusions to eliminate noise, then maps confirmed findings to attack chains showing how individual vulnerabilities chain into full exploitation paths
  6. Pushes every finding to Splunk HEC as a structured battleops:finding event in real time (severity, file, line, CVE ID, risk score, run ID, all queryable)
  7. Posts inline review comments back to the GitHub PR, anchored to the exact changed lines where each vulnerability was found
  8. Anomaly detection runs continuously on the Splunk side: the AI Toolkit's | anomalydetection SPL command flags repositories whose risk scores are statistical outliers and fires a webhook back to BattleOps

Security teams see a live Splunk dashboard: findings by severity, risk score trends per repository, CVE hit rates, and anomaly alerts with links back to the specific PR and run. Developers see inline code feedback without ever leaving GitHub. Both audiences share the same reference per issue. The Splunk event and the GitHub comment are the same finding.

How we built it

BattleOps started in April 2026 as a broader security testing concept. The Splunk integration (HEC, MCP, anomaly detection, and the dashboard) was purpose-built for this hackathon submission in June. Here's the technical journey.

The original architecture: Cloudflare Workers + Durable Objects

The first version ran on Cloudflare Workers with Durable Objects for per-repo state, D1 (Cloudflare's SQLite) for persistence, and the Mastra multi-agent framework for agent coordination. We merged 8 PRs on this stack, building the supervisor-worker agent pattern, forked agent isolation (each scan runs in an isolated instance), permission enforcement, and OpenAPI type safety across 54 security tool implementations.

Why we migrated: the Splunk historical query problem

When we designed the Splunk MCP mid-loop query where the agent asks Splunk for prior findings on the files it's currently reviewing, we hit a wall. D1 couldn't support the queryable findings schema we needed. pr_runs, pr_findings, and pr_cve_hits needed to be JOINable, aggregatable, and available to both the API and Splunk. D1 isn't that. On top of that, Cloudflare Workers' stateless execution model conflicted with the long-running agent pipeline that might spend 30–60 seconds reading files and tracing taint flows.

We migrated the entire stack to Bun + Hono on a Node.js-compatible server in a single day (June 5). The key to doing this without rewriting all agent code was @battleops/cf-compat — an internal shim package that maps the Cloudflare Env interface to real infrastructure: env.KV → Redis, env.DB → PostgreSQL, WorkflowEntrypoint → a local Bun workflow runner. The agent code and imports never changed, but the runtime did.

The AI pipeline

The PR review pipeline went through two major generations. The first used sequential one-shot LLM calls, with scan, classify, analyze, format. The second, which ships now, is a single self-directed tool-calling agent that:

  • Calls repo_read_file to read the full source of files referenced in the diff
  • Calls repo_search_code to trace how a tainted value flows through the codebase
  • Calls splunk_search_context to retrieve historical finding frequency from Splunk
  • Calls skill_load to load MITRE ATT&CK-mapped attack methodologies, so the agent detects the framework in use (React, Django, Rails, etc.) and pre-loads the relevant playbooks

After the primary agent completes, a verifier agent runs adversarially, reading the same code and is specifically instructed to challenge every HIGH/CRITICAL finding. Only findings that survive the challenge are confirmed and pushed to Splunk HEC.

Splunk integration

The integration is genuinely bidirectional. Outbound: findings are pushed to HEC as NDJSON immediately after the verifier completes, sourcetype=battleops:finding. Inbound: mid-analysis, the agent calls the Splunk MCP Server using the splunk_run_query tool over JSON-RPC. This happens before the verdict is formed: Splunk context shapes the AI's confidence scores, not just the metadata. The anomaly detection loop closes the circuit: a scheduled SPL search using | anomalydetection on avg(risk_score) per repo fires a webhook back to BattleOps when it detects outliers.

The Splunk dashboard (splunk/dashboard.xml) and anomaly detection saved search (splunk/anomaly_search.xml) ship with the repo as importable XML.

The web app and GitHub App

The SvelteKit web UI streams run events over SSE in real time, so users can watch the pipeline execute stage by stage. The GitHub App integration required HMAC-verified webhooks, JWT private key auth, installation ID lookup, diff fetching (with fallback to the paginated files API for PRs with more than 300 changed files), and review posting working end-to-end against real repositories. We tested against real PRs, not mocked responses, as you can see in our repo PRs.

Observability

Every agent turn is traced through Langfuse (span hierarchy, token counts, latency per tool call). This turned out to be essential for debugging the verifier agent: you can see exactly which findings it challenged, what reasoning it applied, and what it let through.

Challenges we ran into

Making the MCP query happen before the verdict, not after.

The instinct is: run the AI, then enrich the output with Splunk data. We had to fully invert that. The Splunk MCP query had to be a tool the agent calls mid-reasoning (before it forms a conclusion), so that historical context affects confidence scores, not just metadata labels. Getting the tool-calling loop ordered correctly, and ensuring the agent actually used the historical data rather than ignoring it, took significant iteration and Langfuse traces to debug.

False positive rate.

Early runs over-flagged aggressively. Every SQL query was "SQL injection," every environment variable was "exposed secret." The verifier agent was a direct response to this. Having a second agent whose only job is to argue against the primary agent's HIGH/CRITICAL findings, reading the same code with explicit adversarial instructions, cut noise meaningfully. The combination of primary analysis plus adversarial challenge produces findings that are actually actionable rather than a list that developers immediately dismiss.

The platform migration.

We started on Cloudflare Workers and migrated to Bun/Node mid-build. The migration itself was clean thanks to the @cf-compat shim, but it happened because we had designed ourselves into a corner. D1's query limitations only became visible when we tried to write the SQL that backs the Splunk historical queries. We had to redesign the persistence layer mid-build, which meant the Splunk integration work started the same day the migration landed.

Getting the GitHub App working end-to-end.

Webhook delivery, HMAC verification, private key JWT auth, installation ID lookup, diff fetching, and review posting are all separate GitHub API surfaces with their own edge cases. Large PRs silently truncate the diff. The diff media type changed between API versions. Review comments require exact hunk-position offsets computed from the raw diff, not line numbers. None of this is visible in unit tests. It only showed up against real repositories.

Splunk SPL correctness.

We shipped the anomaly detection search with index=main instead of index=battletest. We had | anomalydetection syntax that required the MLTK add-on instead of native AI Toolkit syntax. We had the MCP server wired to the wrong endpoint format before aligning to the JSON-RPC protocol that app 7931 uses. These were all fixable bugs, but only findable by running the actual integration against a live Splunk instance.

Accomplishments that we're proud of

Splunk as agent memory, not just a log sink.

The MCP mid-loop query is the technical achievement we're most proud of. It makes Splunk an active participant in AI reasoning rather than a passive recipient of AI output. A file with five confirmed high-severity findings in the last 90 days gets treated differently by the agent than a clean file, because the agent knows. That's a genuinely new integration pattern, and it's only possible because of what the MCP protocol enables.

A verified adversarial pipeline.

The two-agent architecture, primary analysis followed by adversarial challenge, produces more calibrated results than any single-pass approach we tried. It mirrors how good human security review works: a finding isn't a finding until it survives challenge. Building that challenge into the pipeline rather than relying on downstream human triage makes the output trustworthy.

End-to-end working against real PRs.

Not a demo with canned data. The GitHub App is installed. Webhooks fire on real PR events. The agent runs, reads real source files, calls real Splunk. Review comments appear on real GitHub PRs anchored to real changed lines. Findings appear in a real Splunk dashboard. The anomaly detection SPL runs on real event data. Every part of the system is tested against real infrastructure.

Attack chain mapping.

Individual findings are useful. Knowing that an injection vulnerability in file A, a missing auth check in route B, and an over-privileged service token in file C combine into a complete account-takeover path is actionable intelligence. The cross-finding attack chain detection does this automatically, and the GitHub review presents it as a connected narrative rather than an unordered list of issues.

A clean migration under pressure.

Moving the entire runtime from Cloudflare Workers to Bun/Node while keeping the agent code intact in a single day using a compatibility shim we wrote ourselves, while under hackathon deadline pressure, and immediately shipping the Splunk integration on top of it, is the kind of execution we're proud of.

What we learned

Splunk's MCP Server fundamentally changes the integration model.

Before MCP, Splunk integrations were mostly one-directional: push events, build dashboards, set alerts. MCP makes Splunk queryable as a tool from inside an agent loop. That's more than an incremental improvement: it's a different relationship between an AI system and a SIEM. BattleOps would not be the same product without it. The historical context that flows into the agent's reasoning is what makes the risk scores meaningful rather than arbitrary.

Adversarial agent pairs outperform single-pass review.

This was the biggest quality insight of the project. A single agent instructed to "be accurate and avoid false positives" is not as reliable as two agents with opposing incentives. The verifier agent doesn't know what the primary agent found, instead it reads the same code fresh and argues against the conclusions. The disagreements are exactly the false positives you want to filter. We'd apply this pattern to any AI output that has high consequences if wrong.

The attack surface of AI-generated code is real and distinctive.

Running BattleOps against real repositories including codebases that are partially AI-generated surfaced patterns we didn't expect. AI-generated code tends to introduce injection vectors in data-boundary handling, skip input validation at internal API boundaries (because the model assumes a trusted caller), and over-trust environment variables. The vulnerability profile is different from hand-written code, and automated review tuned to those patterns catches things human reviewers often miss because the code looks clean at a glance.

Platform decisions made early have compounding consequences.

We chose Cloudflare Workers for good reasons (edge-native, globally distributed, elegant Durable Objects for per-repo state). But D1's limitations didn't become visible until we tried to write the SQL that backs Splunk's historical queries two months later. If we'd started on Postgres, the Splunk integration design would have been simpler from the start. When persistence is a first-class concern (and in security tooling, it always is), choose the storage layer first and let the runtime follow.

What's next for BattleOps

Multi-repo risk correlation.

Every repository currently has its own isolated risk profile. The next step is cross-repo correlation in Splunk, like detecting when the same vulnerability pattern appears across multiple repositories simultaneously. This is the signature of a compromised AI code suggestion propagating at scale, or a shared internal library introducing the same flaw everywhere. Splunk's data model is already structured to support this; it's a SPL search and a UI change away.

Kill chain visualization.

The verifier agent already maps confirmed findings to MITRE ATT&CK techniques and attack chain steps. We want to surface this as an interactive diagram in the Splunk dashboard, showing how individual findings compose into a full exploitation path, with each node linking to the specific PR and source line. The goal is making risk legible to non-technical stakeholders: a CISO should be able to see "this PR opens a path from public input to database admin" without reading code.

Automated triage routing using Splunk alert priority.

Right now all findings go to the same place, that is the developer's PR and the Splunk dashboard. The right behavior is tiered: a CRITICAL finding with confirmed Splunk alert history goes directly to the security team's incident queue; a LOW finding on a file with no history gets a non-blocking developer comment. Splunk has all the data to make this routing decision already.

Broader language coverage.

The current agent handles TypeScript and JavaScript well. Python (Flask, Django, FastAPI), Go, and Rust have different vulnerability idioms, so injection in Python ORM queries looks nothing like it does in TypeScript, and Rust's memory safety eliminates whole vulnerability classes while introducing others. Extending the skill library and stack-detection logic to handle these runtimes is the next coverage milestone.

An SME-tier hosted offering.

The organizations most exposed, that is small teams shipping fast with AI-assisted development, no dedicated security staff are the least likely to run their own Splunk instance and self-host BattleOps. A managed tier with shared Splunk infrastructure, pre-configured HEC and MCP, and a one-click GitHub App install would put this tooling in front of exactly the businesses described in the inspiration. That's the long-term product direction.

Built With

Share this project:

Updates