MR. Robot Adversarial

Inspiration

About the project

MR. Robot Adversarial is an autonomous AI cyber-defense agent built for the SANS FIND EVIL! Hackathon 2026. The core idea is simple: if AI-assisted adversaries can move at machine speed, defenders need a triage loop that is fast, auditable, and harder to fool than a single LLM prompt.

The project triages suspicious artifacts through a four-stage pipeline:

Deterministic scanner sweep using skill_scanner, ioc_scanner, scan_yara, and secrets_detector.
MR. Robot triage agent using an LLM workflow with input gathering, attack-surface mapping, security checklist review, verification, and pre-conclusion audit.
Heterogeneous falsifier using a different model family to challenge the first verdict.
Rule-based synthesizer with $\tau = 0$, so the final verdict is not decided by an LLM judge.

The output is a verdict, confidence score, rationale, MITRE-style context, and a SQLite audit trail that records each stage of the run.

## Inspiration

The inspiration came from the asymmetry between modern AI-accelerated attackers and human incident responders. Tools like autonomous exploit generators and multi-agent offensive systems can operate in minutes, while defenders still lose time switching between scanners, logs, CLI tools, and manual reasoning.

I wanted to build the smallest defensible slice of an autonomous SOC analyst: not a system that claims to solve all cybersecurity, but one that can reliably answer a focused question:

Given one suspicious artifact, can an agent scan it, reason about it, challenge itself, and leave behind enough evidence for a human to trust or reject the verdict?

That led to the central design principle: evidence integrity over model confidence.

## What it does

MR. Robot Adversarial analyzes files such as Python scripts, JavaScript packages, shell scripts, YAML, CI/CD configs, prompt-injection artifacts, cloud abuse samples, and benign framework code.

It detects indicators such as:

C2 URLs and suspicious domains
reverse shells and bind shells
credential theft and hardcoded secrets
malicious package behavior
prompt injection and tool-output injection
CI/CD and supply-chain abuse
suspicious Kubernetes, Docker, and cloud patterns

It also tries to reduce false positives by recognizing framework-safe patterns, such as Django ORM parameterization, auto-escaped templates, and benign test fixtures.

## How I built it

The project is implemented in Python 3.11 with a modular scanner and orchestration architecture.

The main components are:

triage_orchestrator.py — coordinates scanner sweep, LLM triage, falsifier review, self-correction, and final synthesis.
agents/mr_robot/triage.py — LLM-powered triage agent with a structured security-review workflow.
triage_falsifier.py — adversarial reviewer that challenges the initial triage result.
scanners/ — deterministic scanners for skills, IOCs, YARA rules, and secrets.
prompt_injection_defense.py — wraps untrusted file contents in a sentinel boundary and detects prompt-injection attempts.
execution_logger.py — writes every stage to SQLite WAL for auditability.
mcp_server.py — exposes the system as MCP-compatible tools.

The architecture intentionally separates responsibilities. LLMs can analyze and challenge, but the final verdict is produced by a deterministic synthesizer. In other words, the LLMs provide evidence, but the rule-based orchestrator owns the decision path.

The heterogeneity check is based on the idea that two agents from the same model family can reinforce the same mistake. The system tracks architectural distance $\Delta A$ and treats same-family review as a kinship-lock risk. The target design is:

$$ \tau = 0 $$

for the final synthesizer, meaning no LLM directly controls the final verdict.

## Challenges I faced

The hardest challenge was not getting an LLM to produce security-sounding output. That part is easy. The hard part was making the output auditable and falsifiable.

Major challenges included:

False positives: Early scanner rules were too broad and flagged safe framework code. I tightened rules around Django ORM, parameterized SQL, Kubernetes manifests, and benign fixtures.
LLM sycophancy / kinship lock: A second LLM from the same family can agree with the first model for the wrong reason. I added a heterogeneous falsifier and tracked $\Delta A$ between propagator and auditor.
Prompt injection inside candidate files: Since the file under review is untrusted input, I added sentinel wrapping and detection for role markers, tool forgery, delimiter attacks, jailbreak strings, and schema hijacking.
Auditability: I had to make every stage reconstructable: scanner output, triage verdict, falsifier status, self- correction, confidence, duration, and final route.
Demo reliability: The final video was re-rendered as a clean 1080p/30fps artifact with sanitized paths and evidence pulled from the repo.

## What I learned

The biggest lesson was that autonomous defense needs architecture-level guardrails, not just better prompts.

I learned that:

A rule-based synthesizer is more trustworthy than asking an LLM to judge another LLM.
False-positive reduction needs framework awareness, not just more signatures.
Every agentic security system needs a durable audit trail.
Prompt injection defense must treat the analyzed artifact as hostile data, not as text to be blindly pasted into a prompt.
Honest reporting matters: the project reports both strong internal accuracy and a lower CyberSOCEval sub-baseline result instead of hiding weak spots.

## Results

The internal evaluation uses 173 labeled samples:

135 malicious samples
38 benign samples

Results from docs/accuracy_report.json:

Metric Result ━━━━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━━━━━━━━━━━━━━━━ Accuracy 99.42% ───────────────────── ─────────────────────────── Precision 99.26% ───────────────────── ─────────────────────────── Recall 100% ───────────────────── ─────────────────────────── F1 0.9963 ───────────────────── ─────────────────────────── False-positive rate 2.63% ───────────────────── ─────────────────────────── Confusion matrix TP=135, FP=1, TN=37, FN=0

The single known false positive is documented: benign_corpus/typescript_dto.ts contains an AWS-key-shaped fixture string and is flagged by the secrets detector.

The project also includes a public benchmark sanity check on a CyberSOCEval malware-analysis subset: