FreshGuard-Agent

Inspiration

In enterprise systems, when a data pipeline silently breaks, nobody finds out until an executive opens a dashboard showing last week's numbers. The frustrating part is that all the metadata needed to catch this failure—freshness, lineage, and ownership—is already sitting in DataHub. It just lacks an autonomous system to act on it. I wanted to build an agent that doesn't just read data, but actually does the "real work" of investigating and filing the ticket so a human engineer doesn't have to.

What it does

FreshGuard-Agent is a fully autonomous DataOps agent. Given a single generic prompt ("Check my pipelines for stale data"), it runs a complete diagnostic loop:

1. Discovers: Scans DataHub for datasets that have breached their freshness SLA.
2. Diagnoses: Walks the upstream lineage breadth-first to find the deepest stale ancestor, distinguishing between a local failure and an inherited one.
3. Acts: Files a native Incident directly into DataHub's catalog against the affected dataset, explicitly naming the root cause in the incident description. ## How we built it he application is built in Python with a Streamlit frontend. To prioritize data privacy, the AI brain runs entirely locally using Qwen3:8B via Ollama on an Apple M4 Pro. I orchestrated the agent using LangChain, writing custom tools that allow the LLM to interact directly with DataHub's metadata graph using the DataHub Python SDK and Rest Emitter. ## Challenges we ran into A big challenge was actually testing the agent locally since the data doesn't change on its own. The local DataHub Docker setup uses Kafka, which was too slow at updating timestamps for a live demo. To fix this, I wrote a Python script (seed_data.py) to manually inject fake timestamps so the agent actually had a stale database to find. I also ran into an infinite loop bug because some tables in the sample data weirdly point to themselves as dependencies, so I had to write custom code to stop the agent from getting stuck.

Accomplishments that we're proud of

I am extremely proud of getting a local 8B model to consistently execute a complex, three-part tool chain without hallucinating URNs or losing context.

Beyond the hackathon project itself, I am proud to have extracted the core LangChain incident-filing tool I built and contributed it upstream. It was successfully merged into the official acryldata/mcp-server-datahub repository (PR #180) as a reusable example for the community. During this process, I also identified and patched a bug in the emitter logic that was returning false-success responses for invalid URNs.

What we learned

I gained deep, hands-on experience manipulating enterprise metadata graphs. I learned how to bind custom programmatic tools to LLM reasoning loops, handle deterministic state staging for AI testing, and write production-grade API interactions using DataHub's Rest Emitter.

What's next for FreshGuard-Agent

The immediate next step is implementing incident deduplication. Currently, the agent files a new incident on every run; it needs logic to check if an active incident already exists for a specific dataset before emitting a new proposal. I also want to expand the toolset to allow the agent to detect and diagnose data volume anomalies alongside freshness drops.

Built With

• datahub
• docker
• langchain
• ollama
• python
• qwen
• streamlit