The Data Guardian

• 

  Thumbnail

• 

  Dashboard

• 

  Agent Interface

Inspiration

Data privacy compliance is a nightmare for modern organizations. While companies spend millions on "static" security tools and perimeter defenses, data leaks often happen in the messy, human reality of support tickets, chat logs, and internal notes. Traditional regex-based tools are noisy and lack context—they flag a user saying "I forgot my password" just as aggressively as someone pasting "My password is hunter2", drowning security teams in false positives.

I was inspired by the "Automate messy internal work" track. I wanted to build an agent that acts like a human compliance officer: one that can instantly scan massive datasets, understand the context of a potential leak, consult corporate policy, and take reliable action to fix it before an audit is required.

What it does

The Data Guardian is an autonomous AI agent built with Elastic Agent Builder that proactively monitors your data streams for PII (Personally Identifiable Information) leaks and remediates them in real time. It transforms a database from a passive storage unit into a self-healing system.

It operates in a continuous, four-step loop:

1. DETECT: It uses high-speed ES|QL tools to instantly filter thousands of documents for high-risk keywords, finding the needle in the haystack.
2. REASON: It uses an LLM to distinguish between safe contexts and genuine threats. It goes a step further by querying a Corporate Policy Vector Store (RAG) to cite specific rules (e.g., "Section 4.2 of PCI-DSS") to justify its decisions.
3. ACT: If a genuine leak is detected, the Agent triggers a predefined Elastic Workflow to surgically redact the sensitive data from the document in the source index.
4. REPORT: Every action is logged to an immutable audit index that powers a real-time, CISO-ready Kibana dashboard showing "Active Threats Neutralized."

How I built it

I built The Data Guardian entirely natively on Elastic Cloud Serverless, leveraging the full power of the new Agent Builder ecosystem:

• Agent Builder: Served as the core orchestrator, containing the system prompt that defined the "Compliance Officer" persona and managing the tool-calling logic.
• ES|QL (Tool): I built a custom scan_support_tickets tool using ES|QL to efficiently query the index and extract the internal _id of suspicious documents.
• Elastic Workflows (Tool): I built a redact-pii-ticket workflow that the Agent can invoke to safely update the Elasticsearch index with the sanitized text, ensuring the LLM doesn't have raw write access.
• Retrieval-Augmented Generation (RAG): I created a company-policies index containing compliance documentation that the Agent searches to ground its reasoning.
• Kibana Dashboards & Lens: Used to visualize the output of the Agent's actions from the audit-logs index.

The agent has no write privileges on the index; the workflow is the only write path.

Challenges I ran into

The biggest challenge was bridging the gap between the Agent's reasoning and executing the state change. Initially, the Agent struggled to pass the correct identifier to the Redaction Workflow.

I learned that standard queries return the business ID (e.g., "Ticket-1002"), but the update workflow required the internal Elasticsearch _id. I solved this by diving deeper into ES|QL and using the METADATA _id command in my scanning tool, allowing us to capture the internal ID and pass it directly into the Workflow parameters via the Agent's context window.

Accomplishments that I'm proud of

I am incredibly proud that I built an agent that takes reliable action. It is very easy to create a chatbot that suggests redaction; it is much harder to build an autonomous system that safely modifies the database state in accordance with policy.

Seeing the Agent autonomously differentiate between a false positive and a real credit card number, cite a specific corporate policy as justification, and safely carry it out—where the agent decided, and a separate workflow with narrowly scoped permissions executed the replacement to [REDACTED]—was my major "Aha!" moment.

What I learned

I learned that context is everything in automation. LLMs are powerful not just for generating text, but for making nuanced decisions that traditional programmatic rules cannot. Furthermore, I learned the immense value of keeping the entire agentic loop—from data ingestion to search, reasoning, and workflow execution—within a single, unified platform like Elastic Cloud.

What's next for The Data Guardian

This MVP proves the concept of self-healing data. My next steps include:

• Restricted-Access Audit Indexing: Implement an encrypted, access-controlled audit vault to preserve raw, pre-redaction text before applying redactions, preventing permanent destruction of forensic evidence required for regulatory audits.
• Semantic Policy Retrieval via ELSER: Upgrade policy lookup from basic search to Elasticsearch semantic search (semantic_text / ELSER) to accurately retrieve and cite policies across paraphrased, domain-specific, or multilingual text.
• "Human-in-the-Loop" Workflows: For low-confidence detections, pause execution and dispatch an interactive Slack notification to security officers for one-click approval before mutating data.
• Multi-Modal Scanning (OCR): Integrate OCR tools so the Agent can extract and inspect text embedded in PDF attachments and uploaded screenshots within support tickets.
• External Integrations via MCP: Package the Agent as a Model Context Protocol (MCP) server to enable direct integration into CI/CD pipelines for pre-merge secret and PII scanning.

Built With

• elastic-agent-builder
• elastic-cloud-serverless
• elastic-workflows
• elasticsearch
• esql
• kibana
• lens
• rag