ContextOps

Catalog health ≠ data health. A DataHub agent that A/B-audits metadata vs SQL freshness on nyc-taxi, proves silent staleness, then writes Incidents, Documents, and tags back into the graph.

Comment

Inspiration

Modern data platforms trust catalog signals, ingestion timestamps, pipeline “success,” green metadata. But DataHub’s own nyc-taxi planted-staleness sample shows a failure mode teams actually fear: staging and mart can be days behind raw while the catalog still looks fine. Dashboards and downstream models quietly rot.

We built ContextOps for the hackathon challenge Agents That Do Real Work: not another chat-over-metadata demo, but an agent that reconciles two sources of truth (catalog vs data plane), then writes results back so the next human or agent inherits the knowledge.

What it does

ContextOps runs a dual-truth silent freshness audit on the official nyc_taxi_pipeline dataset:

  • Arm A (metadata-only): DataHub search / catalog signals → typically PASS after ingest
  • Arm B (context + SQL): real MAX timestamps on SQLite → FAIL when staging/mart lag raw
  • Aha: Arm A PASS + Arm B FAIL on the same assets
  • MCP depth: official get_lineage_paths_between blast radius (raw → staging → mart)
  • Write-back: Incidents, Context Document runbook, staleness tags
  • Agent citizenship: Agent Registry emit, or honest Document inventory fallback on OSS Core

Primary path: Gemini tool-calling via POST /agent/silent-freshness. Deterministic fallback (no LLM): POST /audit/silent-freshness. Judge UI at Next.js shows Arm cards, Aha, MCP lineage paths, tool trace, and DataHub deep links.

How we built it

  • Backend: FastAPI + acryl-datahub SDK + Google Gemini function calling
  • Official agent stack: mcp-server-datahub (search, lineage, paths, queries, schema, mutations) and datahub-agent-context (build_langchain_tools)
  • Data plane: official planted-staleness SQLite (nyc_taxi_pipeline.db)
  • Frontend: Next.js App Router console (TanStack Query, Zustand, Tailwind)
  • Evidence pack: examples/ golden outputs (ab_metadata_only.json, incident_*.json, runbook.md, MCP/ACK inventories)
  • One-shot demo: make quickstart-demo (after local DataHub quickstart + PAT)

Repo: https://github.com/ChiJian28/ContextOps (Apache 2.0)

Challenges we ran into

  • OSS Core vs Cloud: native Agent Registry entities are not always available on quickstart, we implemented an honest Document fallback instead of faking Cloud-only APIs.
  • Catalog ≠ data: proving the planted failure required wiring real SQL checks, not metadata alone.
  • Gemini tool payloads: protobuf / RepeatedComposite args broke FastAPI JSON until we sanitized tool results.
  • Browser networking: LAN access to 127.0.0.1 backend failed until we added a Next.js /backend-api rewrite.
  • Scope discipline: staying OSS-reproducible (no Cloud Private Beta dependency) while still using MCP + ACK deeply enough for Integration Score.

Accomplishments that we're proud of

  • A reproducible PASS vs FAIL Aha on official DataHub sample data, not a mock.
  • End-to-end governed write-back into the graph (Incidents + Documents + tags).
  • First-class MCP + ACK service surfaces (not buried behind generic HTTP).
  • Path-aware impact with get_lineage_paths_between, visible in the UI.
  • A judge-friendly console + examples/ pack so reviewers can inspect quality without a full local run.

What we learned

  • Agents need complete context and sometimes must arbitrate catalog context vs data-plane truth.
  • DataHub’s MCP tools and Agent Context Kit are the right integration surfaces for “agents that do real work.”
  • Write-back and agent citizenship matter as much as detection: the handoff is the product.
  • Honest OSS fallbacks (Document inventory) beat claiming Cloud-only features we couldn’t run.

What's next for ContextOps

  • Hosted judge demo (one-click URL) on top of the current local quickstart path
  • Optional Actions-driven re-audit on metadata change events
  • Optional healthcare selective-impact vignette
  • Deeper governed proposal workflows where Cloud/Core support allows
  • Continuous freshness monitoring as a production ContextOps service

Built With

Share this project:

Updates