FlowJury

FlowJury is a agent that investigates complex data pipelines and recommends safe actions to reduce waste and protect critical workflows.

Comment

Inspiration

Large organizations operate thousands of data pipelines. Some quietly waste compute, duplicate existing work, fail repeatedly, or run more frequently than necessary. Others appear unused while secretly supporting customer deliveries, machine-learning systems, or regulatory processes.

The difficult part is not finding pipelines with low usage. The difficult part is proving whether changing them is safe.

I created FlowJury to automate this investigation. Instead of presenting engineers with another dashboard, FlowJury gathers evidence, investigates hidden dependencies, reasons through business policies, and produces a reviewable recommendation.

What FlowJury does

FlowJury is a LangGraph agent that gives every pipeline a safe, explainable lifecycle verdict.

It can recommend:

  • KEEP for pipelines with active or hidden consumers
  • PROTECT for compliance, privacy, and retention workflows
  • RUNAWAY for executions far beyond their normal duration
  • DOWNSHIFT for pipelines running more frequently than necessary
  • TRIM for pipelines producing unused data
  • FIX/FOLD for repeatedly failing workflows
  • KILL or REDUNDANT when retirement is supported by sufficient evidence
  • UNKNOWN when the evidence is incomplete

FlowJury never automatically disables a pipeline. It creates evidence-backed proposals for human approval.

How I built it

FlowJury uses DataHub as its organizational context and system of record. Through DataHub and its Agent Context Kit, the agent investigates:

  • Pipeline and dataset lineage
  • Usage statistics
  • Ownership and domains
  • Governance and compliance tags
  • Pipeline run history
  • Executable DAG evidence
  • Downstream blast radius

The agent is orchestrated with LangGraph using three focused nodes.

  1. The Supervisor examines the available evidence, recalls relevant memory, and selects the business skills or tools required for the current pipeline.
  2. The Executor performs those investigations and validates the evidence.
  3. The Skeptic Review independently challenges high-risk KILL and REDUNDANT proposals.

When evidence is missing, execution returns to the Supervisor for another focused investigation. This produces a bounded agentic loop rather than a fixed scoring formula.

Business logic is stored in versioned skills. The agent loads only the specialist policies relevant to the situation, such as compliance protection, external delivery detection, runaway compute, or redundancy analysis.

FlowJury also has durable investigation memory. It fingerprints the current evidence and recognizes pipelines as FIRST_SEEN, CHANGED, or UNCHANGED. Previous verdicts provide context, but current DataHub evidence always takes priority.

Blast radius is calculated from DataHub lineage rather than guessed by the language model. Completed proposals—including confidence, evidence, risks, applied skills, and next actions—can be written back into DataHub.

Challenges I faced

The biggest challenge was that zero usage does not mean zero value. A dataset may have no warehouse queries while still supporting HubSpot, Kafka, an online feature store, an external API, or a regulatory process.

Another challenge was balancing agent autonomy with safety. The agent needed freedom to choose its investigation, but destructive recommendations required stronger controls. I introduced evidence citations, bounded execution, deterministic blast-radius analysis, structured validation, and an independent Skeptic Review.

LLM variability was also a challenge. Early versions could produce different verdicts for unchanged evidence. I reduced sampling variability, strengthened proposal validation, and made incomplete investigations fall back visibly to UNKNOWN instead of producing unsupported decisions.

Finally, I created 18 realistic enterprise pipeline scenarios to test situations that simple usage-based rules could easily misclassify.

What I learned

I learned that metadata becomes significantly more valuable when an agent can investigate relationships instead of retrieving isolated records.

I also learned that a trustworthy agent needs more than reasoning. It needs evidence, bounded execution, validation, durable memory, and the ability to admit uncertainty.

Most importantly, DataHub can act as shared organizational memory. FlowJury reads that knowledge, reasons over it, and contributes its findings back so the next engineer—or agent—does not have to restart the investigation.

What's next

Future improvements could include scheduled change monitoring, approval workflows, cost-platform integrations, collaborative specialist agents, and richer DataHub-native proposal experiences.

FlowJury’s goal is simple: give every pipeline a fair, explainable verdict before engineers spend money maintaining it—or take the risk of removing it.

View the FlowJury source code on GitHub

Built With

  • agent
  • agents
  • ai
  • context
  • data
  • datahub
  • docker
  • engineering
  • github
  • governance
  • kit
  • langgraph
  • language
  • large
  • lineage
  • metadata
  • models
  • pytest
  • python
  • sdk
  • skill-based
  • sqlite
Share this project:

Updates