DATA LAB

DataHub-grounded agents propose inspectable pipeline changes; humans approve them atomically and replay every decision.

Comment

DATA LAB — Human-reviewed data pipeline agents

DATA LAB is an Electron visual pipeline studio grounded in DataHub schemas, lineage, ownership, quality, and governance metadata. A data engineer starts from a blank workbench, binds a real DataHub dataset to a Data Source card, and asks the agent to investigate or improve the pipeline. Before proposing a change, DATA LAB reads the catalog through the official DataHub MCP server and records the exact evidence snapshot used by the revision.

The agent cannot invoke arbitrary MCP tools or silently change the committed graph. It returns a strict, bounded graph diff made from inspectable roles such as Data Profile, Impact Analysis, Transform, Agent Decision, Human Review, Validation, and Output. DATA LAB materializes the candidate graph separately, evaluates it with independent atomic validation rules, and presents the complete diff in a large review modal. Rejection leaves the active graph unchanged; approval creates a replayable SQLite checkpoint.

Verified DataHub OSS scenario

Using the official ecommerce showcase, DATA LAB resolves the real dbt order_details dataset, reads 40 schema fields, identifies 12 PII classifications, and observes 18 downstream datasets across Snowflake, dbt, Looker, Power BI, and Tableau. A direct analytics path is blocked because customer contact fields are unprotected. The reviewed correction adds compact profile memory, an impact snapshot, deterministic PII protection, and a Human Review gate before output.

Why DataHub is essential

Without DataHub, the model would have to invent table names, schemas, classifications, owners, and downstream impact. DATA LAB instead calls get_entities, list_schema_fields, and get_lineage through the official DataHub MCP server. Evidence is bounded and sanitized: no raw rows, credentials, authorization headers, or provider secrets are stored.

Human control and reproducibility

  • Blank workbench by default; no simulated healthy result.
  • Strict provider proposal contract and read-only MCP allowlist.
  • Independent atomic safety checks before graph mutation.
  • Human Review notifications and emergency stop.
  • Versioned evidence, accepted/rejected revisions, and replay.
  • Optional governed DataHub writeback behind a separate native confirmation.
  • ChatGPT account connection plus OpenAI, Claude, and Kimi API providers.
  • Local-first SQLite persistence and private diagnostics.

Pre-existing work disclosure

DATA LAB was created during the hackathon. Its XY graph placement algorithm adapts connected-component, layered-topology and barycentric-sweep concepts from my pre-existing Apache-2.0 LABO AI project (https://github.com/Complexity-ML/labo-ai). All DataHub/MCP integration, data-card domain logic, provider contracts, agent review workflow and submission artifacts were created specifically for DATA LAB.

The repository includes complete setup instructions, Apache 2.0 licensing, sanitized MCP traces, a reviewed graph diff, an atomic validation report, final screenshots, an architecture diagram and a timed demo script.

Built With

Share this project:

Updates