WatchTower

Attack surface mapping, powered by DataHub.

Comment

Inspiration

Every company maps how their data flows. They invest in data catalogs, build lineage graphs, and track which systems feed into which dashboards. But nobody was asking the obvious question: if an attacker got access to one table in this graph, how far could the damage spread?

I realized that DataHub's lineage graph — the same metadata that helps data engineers understand dependencies — IS an attack surface map. It shows exactly how sensitive data propagates across platforms, which assets have no governance, and where a single compromise could cascade into dozens of exposed systems. The map was already there. Nobody was reading it as a threat model.

That's what Watchtower does. It takes the metadata you already have in DataHub and asks security questions against it.

What it does

Watchtower is an AI-powered security posture agent that sits on top of DataHub. It reads the metadata graph, reasons over it with an LLM, identifies vulnerabilities, and writes security enrichments back to the catalog — making each scan smarter than the last.

Five security engines powered by DataHub's metadata graph, all writing back to make the catalog smarter.

Breach Simulation

You pick any entity in the catalog — a table, a source system, a data warehouse — and ask: "What happens if this is compromised?"

The engine traverses all downstream lineage from that entity, mapping every system, table, dashboard, and report that receives data from it. For each affected asset, it checks whether it contains PII, whether it's governed, who owns it, and how many hops away it is from the breach source. The result is a blast radius map with a multi-dimensional impact score.

What makes this engine different is attacker personas. The same breach tells three different threat stories depending on who the attacker is — External, Insider, or Supply Chain. Each persona changes the risk weighting: an external attacker cares about downstream propagation to public dashboards; an insider exploits ungoverned assets with broad access; a supply chain attack traces how vendor data cascades through your systems.

→ View breach simulation engine code

Threat Model Concept

Security Posture Score

A single 0-100 score with a letter grade (A through F) representing how well-governed your DataHub catalog is. The score is computed across five dimensions: governance tags, ownership coverage, descriptions, glossary terms, and quality assertions.

The key insight: the better you use DataHub's native governance features, the higher your security score. The score also generates specific suggestions: "Add tags to 5 more datasets (+5 points)" so teams know exactly what to do next.

→ View security score engine code

PII Exposure Scan

The scan engine searches your catalog, uses an LLM to classify every column for sensitivity, then traces downstream lineage multi-hop for each sensitive column. It scores risk at each hop based on governance gaps and produces a ranked list of exposure paths. In write mode, it auto-tags unprotected PII columns, applies glossary terms, and saves a full scan report to DataHub.

→ View PII scan engine code

Compliance Drift Detection

The drift engine captures a snapshot of your catalog's governance state and compares it against the current state over time. It detects regressions: tags that were removed, owners that disappeared, new datasets that appeared with zero governance, PII columns that were added without classification.

The engine then uses an LLM to generate root cause analysis. Not just "this tag was removed" but "this ownership gap was likely caused by team restructuring — recommended process fix: add governance review to pipeline deployment." It diagnoses organizational problems, not just metadata gaps.

→ View compliance drift engine code

Agentic Chat

A conversational interface backed by a LangGraph ReAct agent with access to 15 DataHub tools. You ask questions in natural language — "Where does customer_email flow?" or "Simulate a breach on the customers table" — and the agent chains multiple operations autonomously.

It searches the catalog, traces lineage, inspects entity metadata, runs scans, and applies remediations — all in a single conversation turn. Each tool call is visible in the UI so you can see the agent's reasoning chain.

→ View chat agent code · View 15 DataHub tool definitions

Attacker Personas

Most security tools treat all threats the same way. But the same data tells a completely different risk story depending on who's looking at it.

Attacker Personas

External Breach: "If this source is compromised, where does the data end up?" Highlights downstream propagation to dashboards, BI tools, and broadly accessible platforms. Focuses on what an external attacker could reach.

Insider Threat: "What's unprotected from within?" Highlights ownerless datasets, untagged PII, and assets with no access controls. Shows what an employee with legitimate access could exploit without detection.

Supply Chain: "If a vendor is compromised, how far does it cascade?" Traces upstream sources flowing into your core systems. Maps third-party trust boundaries and the blast radius of a vendor-side breach.

Same lineage graph. Three different questions. Three different risk maps.

The Write-Back Loop

Watchtower doesn't just read the catalog and report findings. It writes security metadata back to DataHub after every scan — PII tags, glossary terms, security descriptions, and full scan reports saved as documents.

This creates a virtuous cycle: each scan enriches the catalog with more context, which makes the next scan smarter (it knows what's already tagged), which improves the security score, which incentivizes teams to maintain governance.

Security Loop

How I built it

Architecture

Agent Framework: LangGraph state graphs for each engine. Every scan is a compiled multi-phase workflow (search → classify → trace → score → remediate → report). The chat interface uses a ReAct agent with tool calling.

DataHub Integration: The datahub-agent-context SDK provides 15 MCP tools — search, get_entities, list_schema_fields, get_lineage, add_tags, add_terms, update_description, save_document, and more. Watchtower uses all of them in both read and write directions. → View DataHub tool wrappers

LLM: OpenRouter with GPT-4o-mini for sensitivity classification, risk narrative generation, and root cause analysis. Low temperature for consistent classification, structured output parsing for reliable integration.

Frontend: React + TypeScript with Vite. TailwindCSS for styling with a custom dark theme (glassmorphism, animated background traces). React Flow (@xyflow/react) for interactive blast radius graph visualization.

Backend: Python 3.11 with FastAPI exposing all engines as REST endpoints. Pydantic models for request/response validation and risk scoring types.

Challenges I ran into

  • DataHub SDK response formats: The datahub-agent-context SDK returns entity data in different structures than expected. Some entities have schema metadata, others don't. I had to build resilient parsing that handles missing fields gracefully.

  • LLM context window limits: With a full catalog (100+ entities), tool call results from search and lineage operations would exceed the 128K token context. I had to trim results to essential fields and cap pagination.

  • Infrastructure on 8GB RAM: Running DataHub (Kafka, OpenSearch, MySQL, GMS) alongside the application pushes memory limits. Kafka health checks frequently time out on constrained machines.

  • Scan performance: A full PII scan with LLM classification, multi-hop lineage tracing, and metadata enrichment takes 1-2 minutes. Balancing depth of analysis vs. response time required careful tuning of batch sizes and dataset limits.

Accomplishments I'm proud of

  • Full DataHub read+write loop. Not a read-only scanner — the catalog genuinely gets smarter after each scan. Tags, terms, and reports persist.

  • Attacker persona modeling. The same breach simulation tells different stories to different stakeholders. A CISO sees external risk; a data team sees governance gaps; a vendor manager sees supply chain exposure.

  • Root cause analysis that diagnoses process failures. "This drift was caused by team restructuring" is a fundamentally different insight than "this tag is missing."

  • Real agentic chat. Multi-step reasoning with visible tool calls. The agent decides what to do, chains 4-5 operations, and explains its reasoning. Not a prompt wrapper.

  • Security score as adoption incentive. Reframing governance as a score that improves when you use DataHub's features is a behavioral design choice that makes the tool self-reinforcing.

What I learned

  • Metadata graphs contain more security signal than most teams realize. Lineage isn't just for debugging data pipelines — it's a threat propagation model.

  • Write-back is the key differentiator. Any tool can scan and report. The ones that enrich the source system create compounding value.

  • Personas change the conversation. Showing a CISO "here's the blast radius" in external-threat framing gets a different reaction than showing a data engineer "here's what's untagged." Same data, different narrative, different urgency.

What's next

  • Additional attacker personas (nation-state, privilege escalation, lateral movement)
  • Temporal anomaly detection — flagging sudden changes in access patterns or lineage connections
  • CI/CD integration — pre-deploy security checks that block pipelines creating ungoverned PII paths
  • Open-source DataHub Security Skill contribution to the datahub-skills repository

Built With

Share this project:

Updates

Submission history