dlp-triage-agent

Auto-learning DLP Alert Triage Agent — multi-stage pipeline with memory that improves from analyst feedback

Comment

The DLP Analyst's Dilemma

Imagine starting every morning with 3,000 security alerts. 2,800 of them are the exact same pattern as yesterday — the same HR manager printing the same payroll report, the same sales rep emailing the same customer list. Yet this morning, just like every morning, each alert runs through an expensive LLM pipeline, burns tokens, takes 30 seconds of someone's attention, and produces the same verdict it produced the last 100 times.

This isn't security operations. This is assembly line work dressed up as threat detection.

Traditional DLP systems are fundamentally broken in one way: they have no memory. They treat every alert as if it's the first time they've ever seen it. An analyst can correct the same false positive 50 times, and the system will happily make the same mistake on attempt #51.

We wanted to change that.

What We Built

DLP Alert Triage Agent is a multi-stage file screening pipeline that learns from every analyst decision. Not by asking analysts to write rules. Not by requiring prompt engineering. Not by any kind of "training mode."

By simply doing their job.

When an analyst clicks Dismiss or Escalate — and that decision contradicts what the pipeline predicted — the system notices. It writes a feedback record. It calls Qwen3.7-plus to extract a memory pattern. And the next time a matching alert comes in, that pattern short-circuits the entire pipeline: zero LLM tokens, ~50 milliseconds, correct verdict.

No "train" button. No manual rule writing. No prompt tuning.

How It Works

The pipeline has four stages, each capable of short-circuiting the rest:

  1. Metadata Analysis — Who, what, when, where. Some alerts die here.
  2. Algorithms — Shannon entropy, structural fingerprinting, encoding detection. No LLM, no tokens. Pure math.
  3. Memory Recall — Learned patterns from past analyst decisions. If there's an exact match, the system returns the cached verdict in ~50ms and stops. This is where the system pays for itself.
  4. LLM Pipeline — Lite LLM → two Specialist Agents analyze independently → if they disagree, structured debate rounds → Risk Scorer produces the final verdict.

Most alerts never reach stage 4. Clean-cut cases exit early, saving 35–75% of token costs.

Self-Organizing Memory

Every learned pattern is a human-readable Markdown file with YAML frontmatter — conditions, confidence score, hit count, creation date, last matched timestamp.

The memory system manages itself:

  • Score-based retrieval: Patterns are sorted by hitCount × 0.35 + confidence × 0.35 + recency × 0.30. The most relevant ones enter the prompt first.
  • Automatic compaction: When the pattern count exceeds 50, the system culls low-confidence stale patterns, merges overlapping ones, and refreshes active patterns.
  • Confidence decay: Drops 5% per week if unmatched. Every correct match adds +0.05. Every contradiction costs -0.20.
  • Exact match bypass: Instantly returns verdict for exact pattern matches — zero context budget consumed.

Multi-Agent Debate

Single-agent analysis has blind spots. A Specialist Agent might flag "doctor sending patient records" as a violation without realizing it's a routine transfer to an authorized research partner.

Our system runs two Specialist Agents independently. If they agree, the verdict stands. If they disagree, they enter up to two rounds of structured debate — each agent reads the other's reasoning and may revise its position. This dramatically reduced false positives compared to our single-agent architecture.

Built on Qwen Cloud

The entire system runs on Qwen Cloud:

  • Qwen3.7-plus for all reasoning — Lite LLM, Specialist Agents, Debate, Memory extraction, Merge decisions
  • Qwen-VL-Plus for screenshot analysis (images compressed via sharp before submission)
  • Deployed on Alibaba Cloud ECS with full CI/CD pipeline

We also use pdf-parse, mammoth, xlsx, and docx for multi-format document support, and better-sqlite3 for durable result caching.

Challenges We Faced

Memory merging was the hardest problem. When two patterns overlap — say, "HR manager prints payroll" and "HR manager prints Q4 payroll" — should the system merge them, keep both, or let the more specific one supersede? Our third iteration finally got it right: a dedicated merge agent powered by Qwen3.7-plus analyzes both patterns and makes the call.

Token budget management was another hurdle. Injecting too many memory patterns eats context and degrades quality. Our score-based sorting ensures the prompt only contains the most relevant patterns.

What's Next

We're working on multi-company memory isolation (patterns learned for Bank Corp shouldn't influence Healthcare+ alerts), real-time alert ingestion via webhook, and a dashboard showing the system's learning trajectory over time.

The Bottom Line

We didn't build a DLP tool. We built a system that gets smarter every time someone uses it — automatically, silently, without asking permission. Your analysts can focus on actual threats instead of repeating the same verdict a thousand times.

The machine learns. The human decides. Everyone wins.

Built With

Share this project:

Updates