MidSphere

Context circuit-breaker for DataHub: audit, selective quarantine, fix artifacts, and MBOM (metadata bill of materials) attestation so agents inherit safer catalog knowledge.

Comment

Inspiration

AI agents that read a data catalog inherit whatever is in the graph. When quality breaks, personally identifiable information (PII) leaks into analytics marts, or freshness service-level agreements (SLAs) slip, the catalog often only records the failure. Downstream agents and Text-to-SQL tools keep consuming unsafe context. MidSphere was built to treat that failure as a circuit-breaker event: stop trusting the bad fork, leave healthy lineage open, ship a remediation path, and leave an attestation the next operator or agent can read.

What it does

MidSphere is a closed-loop context circuit-breaker for DataHub. On a root dataset uniform resource name (URN) it runs five stages:

  1. ContextSentinel: audit findings (quality, PII, freshness) from scenario packs
  2. CascadeAnalyzer: selective blast radius and risk score
  3. CircuitBreaker: advisory quarantine (tags / structured properties when GMS (Generalized Metadata Service) mutations are on; otherwise job-local)
  4. FixArchitect: dry-run SQL/dbt (data build tool) fix path under examples/prs/
  5. MBOMEngine: metadata bill of materials (MBOM) audit attestation under examples/mboms/, plus live save_document when mutations are enabled

Selective halt is a core behavior: for healthcare, mart_billing can be BLOCKED while mart_demographics stays OPEN. The Command Center shows the graph, pipeline stages, Decision rail (Findings / Gate / Attestation), sticky Trip / Lift, and a top-bar GitHub control to the public source. Decision Gate can probe live GET /v1/gate/{urn}. MidSphere Consumer Gate returns BLOCKED for MidSphere-aware clients. We do not claim platform-wide MCP auto-denial of all third-party agents.

When GMS is up, official Model Context Protocol (MCP) tools are called for real (get_lineage, list_schema_fields, and writes when mutations are enabled, including remove_tags on Lift). Live results are annotated in telemetry and blast_radius.live_enrichment; scenario packs still drive findings, risk scores, and selective impact so demos stay reliable. When the catalog virtual machine (VM) is stopped, a Wake GMS control lets a reviewer start it directly from the Command Center. A Timeline control (off by default) replays the closed loop as a narrative scrubber; it is not a live GMS history browser.

Three scenarios ship: Hospital (healthcare quality), Retail (ecommerce PII), Taxi (NYC freshness). All three are backed by real DataHub ingestion recipes from the upstream static-assets repository. Live demo: midsphere.vercel.app. API: Cloud Run. Source (public, Apache 2.0): github.com/adi-IL/MidSphere.

How we built it

  • Frontend: React 18, Vite, TypeScript, Tailwind, React Flow (minimal monochrome Command Center on Vercel)
  • Backend: FastAPI + Pydantic closed-loop API on Google Cloud Run
  • Primary workforce: Google Antigravity SDK for Python (pip install google-antigravity; Agent + LocalAgentConfig from google.antigravity) with five MidSphere stage tools on Vertex Gemini
  • Catalog: DataHub GMS + official mcp-server-datahub via uvx (real tool names only: get_lineage, list_schema_fields, add_tags, save_document, and related allowlisted tools)
  • Honesty contract: execution_mode is ground truth (antigravity_closed_loop | demo_closed_loop | live_mcp); mutations_applied only when GMS + mutations are enabled; the offline path states plainly that telemetry is simulated
  • Artifacts: scenario-aligned MBOMs, dry-run diffs, and sample telemetry under examples/
  • Open-source skills: proposed datahub-circuit-breaker and datahub-quality-circuit (datahub-skills#88)

Challenges we ran into

  • Live GMS from Cloud Run. Laptop localhost is unreachable from Cloud Run, so a live catalog needs a Google Compute Engine (GCE) quickstart VM, firewall rules, and a re-point whenever the VM IP changes. After a VM resume, DataHub containers can stay stopped. We solved this with the Wake GMS button in the Command Center: it restarts the containers and re-points the Cloud Run environment before health goes green. [Note: This button only appear when GMS goes to sleep.]
  • Tag bootstrap before writes. Tag URNs must exist before MCP add_tags; a missing bootstrap step fails mutations silently from a demo perspective.
  • Build-time env on Vercel. VITE_API_BASE_URL is baked into the single-page application (SPA) at build. A Sensitive-marked env injected a literal placeholder and production silently fell back to offline demo mode on relative /v1/* paths.
  • Token durability in continuous integration (CI). Vercel CLI OAuth tokens expire and break GitHub Actions deploys. We moved to a durable account token plus a weekly keepalive workflow.
  • Honest degradation. The offline path must say what it is: simulated telemetry, job-local quarantine, mutations_applied=false, never an imitation of live MCP writes.
  • Selective halt consistency. Graph, gate, and quarantine set must agree, so the open fork stays visible while the impacted fork is blocked.

Accomplishments that we're proud of

  • End-to-end closed loop with selective blast radius across three scenarios (Hospital, Retail, Taxi)
  • Scenarios backed by real DataHub ingestion recipes from the upstream static-assets repository
  • Live write-back through the official MCP server: trip applies add_tags, lift calls remove_tags when GMS and mutations allow
  • Honest status model (Live / GMS / Sample) with a cold-start boot hint; the UI never presents a sample as a live job
  • Consumer Gate as explicit MidSphere policy, probeable at /v1/gate/{urn}
  • Wake GMS control: a judge can start a stopped catalog VM from the Command Center
  • Judge-readable offline artifacts under examples/ that match backend/agents/scenarios.py
  • Upstream skills PR datahub-skills#88 (datahub-circuit-breaker, datahub-quality-circuit)
  • Public Apache 2.0 monorepo with path-filtered CI/CD to Vercel and Cloud Run via Workload Identity Federation

What we learned

  • Context graphs need write-back paths that agents can honor, not only search and chat
  • Selective impact beats nuclear quarantine: blocking one mart while its sibling stays open is what makes a circuit breaker useful on multi-mart lineage
  • Demos are judged on trust. Honest labels (Sample vs Live MCP) matter as much as the agent loop
  • A deterministic fallback is what makes an agent demo survivable when the LLM path fails
  • Operator reliability is product surface: GMS scheduling, token hygiene, and build-time env decide whether a judge sees a product or an error page

What's next for MidSphere

  • Stronger live warehouse grounding of findings when GMS queries are available
  • Optional GitHub PR open (beyond dry-run paths) when credentials allow
  • Broader consumer adapters that honor advisory quarantine tags
  • Light theme and Pro density mode for operators who want always-on telemetry

Built With

  • datahub
  • datahub-mcp-server
  • docker
  • fastapi
  • gemini
  • github-actions
  • google-antigravity-sdk
  • google-cloud-run
  • pydantic
  • python
  • react
  • react-flow
  • tailwind-css
  • typescript
  • uv
  • vercel
  • vertex-ai
  • vite
Share this project:

Updates