Latch

LATCH turns an external AI-training dataset transfer into a governed release: DataHub context, BigQuery proof, human approval, safer output, and a verifiable passport.

Comment

Inspiration

Imagine a mobility company preparing customer-support conversations for an external model-adaptation provider.

The dataset looks reasonable: questions about charging sessions, useful agent answers, issue categories, and language labels. An engineer could copy it into a new cloud location in minutes.

But “the file looks harmless” is not the same as “this exact release is permitted and ready.”

A direct email column may be obvious; an account number embedded inside a message is less obvious. A field named ai_training_allowed matters only if the organization has defined what it means. A clean-looking view may still descend from a governed source. Some rows may not be eligible for model training. A transformation can remove risk and also destroy the multilingual or issue coverage that made the dataset useful. Finally, an approval in chat does not prove which version was actually published.

That is the gap LATCH addresses. It turns the export from a copy operation into a governed transaction grounded in the organization's own data context, executed against the real warehouse, separated by human approval gates, and closed with durable evidence.

One release, step by step

Step Without LATCH With LATCH How DataHub helps
1. State the goal “Send support data to the model team” is buried in a ticket or chat. The purpose, destination, output, and validity window may not travel together. The requester declares the exact DataHub asset, BigQuery table, external destination, fine-tuning purpose, requested fields, utility floor, and release location. DataHub gives LATCH a stable asset identity instead of a filename that can be copied or renamed.
2. Find the responsible people Engineers ask around to learn who owns the table and who can approve its use. LATCH requires the configured owner and records the people who approve consequential steps. Missing required ownership stops contract compilation. DataHub supplies the recorded dataset owner and other governance context.
3. Understand the fields A column name such as ai_training_allowed can be misunderstood, while innocent-looking fields may carry classifications known only to the catalog. LATCH freezes schema, classifications, glossary meaning, tags, and domain context into a versioned snapshot. DataHub explains what fields mean to this organization, not just their SQL types.
4. Trace where the data came from A final view can look clean even when a field came unchanged from a sensitive upstream table. Manual lineage review is easy to skip. Required table and field lineage become part of the release contract. If required lineage is absent, LATCH stops instead of guessing. DataHub can show that a field in the release source was derived from, or passed through from, an upstream governed source.
5. Choose the rules A reviewer assembles an informal checklist that may differ from the previous release. LATCH matches a versioned policy and deterministically compiles the exact checks, required context, approval gates, allowed transformations, and publication rules for this request. DataHub classifications, glossary terms, ownership, and lineage materially determine whether the policy can be compiled and which fields require action.
6. Inspect the actual data Sampling a few rows may miss identifiers embedded in free text, ineligible records, or small cohorts. Sending raw rows to a chatbot creates another data boundary. BigQuery executes allowlisted, parameterized, byte-capped aggregate checks for configured text patterns, eligibility, cohort size, row count, required fields, category coverage, and duplicate rate. Raw rows stay in BigQuery. DataHub tells LATCH what to inspect and why; BigQuery measures whether the configured condition exists in the actual rows.
7. Decide whether to stop A review can be marked green because a tool failed, a result was assumed, or an AI sounded confident. BLOCKED exists only when a mandatory deterministic check ran, evidence was persisted, and the observation violated the contract. A BigQuery error is an error, never a pass. The failed evidence remains attached to the DataHub-grounded contract that caused the check.
8. Design a safer release An engineer writes one-off SQL and hopes it removes enough risk without ruining the data. LATCH composes two to four policy-allowed alternatives, such as removing direct fields, redacting configured identifiers in text, filtering ineligible rows, grouping rare categories, or suppressing small groups. DataHub's field meaning and classification constrain which transformations are relevant; Gemini may explain only the already allowed templates.
9. Compare trade-offs Reviewers compare intentions rather than executed outcomes. Every alternative is dry-run and previewed in BigQuery. LATCH shows real retained rows, retention, required-field preservation, safety failures, job IDs, query hashes, and estimated bytes only after execution. DataHub keeps the comparison tied to the governed source and its context; it is not an isolated SQL experiment.
10. Make the human choice “Looks good” in chat is disconnected from the SQL and metrics that were reviewed. A human selects an eligible measured plan and approves the exact transformation version. Gemini cannot approve or create a proof state. The approval is bound to the contract derived from DataHub context.
11. Verify independently Teams may assume that intended transformation equals successful output. LATCH materializes a separate release-scoped candidate, then runs the same Safety Contract and the Dataset Utility Contract again. A failed candidate remains failed. The source identity, required lineage, policy context, and limitations stay attached during re-verification.
12. Approve and publish A final copy can overwrite an old table or diverge from the reviewed candidate. A second human approval authorizes publication. BigQuery copies the verified candidate to a versioned target with WRITE_EMPTY, then LATCH verifies that the target exists. DataHub provides the source identity used to create the final source-to-output lineage.
13. Preserve the proof Later, nobody can reconstruct the exact request, policy, SQL, evidence, or approvals. LATCH seals a Release Passport containing the request, context, contract, evidence, approvals, transformation plan and SQL, lineage evidence, reports, and SHA-256 manifest. The passport preserves the exact DataHub snapshot and evidence hashes that informed the release.
14. Close the catalog loop The new dataset exists outside the catalog, so the next person or agent repeats the investigation. LATCH writes the output's release properties plus table- and retained-field lineage into DataHub, reads them back, stores a receipt, and reaches COMPLETE only after required closure succeeds. The governed output becomes discoverable context for the next person or agent.

DataHub in one sentence

DataHub tells LATCH what the dataset means inside the organization and where it came from; without that context, LATCH refuses to pretend that row scanning alone is a release decision.

System Actual responsibility
DataHub Canonical asset context: schema, ownership, tags, glossary terms, domain, table lineage, field lineage, and the governed output written back after publication.
BigQuery Deterministic execution: baseline checks, remediation previews, candidate materialization, safety/utility re-verification, and immutable-style publication.
Gemini Bounded analysis: inspect normalized context and aggregate evidence, compare measured plans, and produce concise explanations.
LATCH Controlled release workflow: compile policy into a contract, enforce state transitions, bind approvals, persist evidence, seal the passport, and coordinate writeback.
Human reviewer Consequential authority: approve checks, choose the transformation, and approve final publication.

Why not upload the table to a general-purpose AI assistant?

A general-purpose model can reason about the rows it is shown, but it does not automatically possess the organization's canonical ownership, glossary meaning, classifications, lineage, destination-specific policy, warehouse execution identity, approval history, or durable release record. It can also be confidently wrong.

LATCH uses Gemini where language reasoning helps, but keeps proof outside the model.

Proof states come from deterministic, versioned contracts and recorded execution. Raw rows remain in BigQuery. Approvals are explicit. The published output is verified, and the decision is preserved for the next reviewer or agent.

What it does

LATCH implements one complete vertical: releasing to an external LLM fine-tuning destination.

  1. Captures a precise release request and declared trust boundary.
  2. Discovers the official DataHub MCP tool inventory at runtime.
  3. Resolves the source asset, schema, ownership, classifications, glossary terms, domain, table lineage, and field lineage.
  4. Compiles that snapshot with the versioned external-llm-finetuning-release policy into a deterministic Release Contract.
  5. Requires human approval before the configured BigQuery evaluation.
  6. Runs bounded aggregate checks; raw warehouse rows are not returned to the API or Gemini.
  7. Enters BLOCKED only from failed persisted evidence.
  8. Builds and executes multiple governed remediation previews.
  9. Lets a human select and approve an eligible measured plan.
  10. Materializes a separate candidate and re-runs both safety and utility checks.
  11. Requires a second human approval before publishing a versioned BigQuery table.
  12. Seals a hash-verifiable Release Passport.
  13. Writes release properties plus table/field lineage to DataHub and reads the result back before COMPLETE.

The application starts with no releases or precomputed proof. The four tiny synthetic BigQuery scenarios are real input infrastructure only; every request, job, finding, plan metric, approval, candidate, passport, and writeback is created through the production workflow.

How we built it

LATCH is a modular monolith with a static Next.js interface and a FastAPI control plane. The domain layer owns the release state machine, policy compiler, evidence model, approvals, and passport. Integrations sit behind explicit adapters.

  • The DataHub MCP client discovers the runtime tool inventory, maps official tools to required capabilities, resolves context, normalizes the returned graph data, and fails clearly when required capabilities or context are missing.
  • The DataHub Python SDK seeds only metadata for the tiny synthetic sources and performs synchronous output-property and table/field-lineage writeback. LATCH reads the aspects back and verifies them.
  • The BigQuery adapter validates identifiers and source/output allowlists, parameterizes detector values, dry-runs queries, applies a maximum-bytes-billed ceiling and timeout, records job IDs and query hashes, materializes candidates, and publishes with WRITE_EMPTY.
  • The Gemini analyst uses structured JSON output and four fixed read-only tools. Its step count and retries are bounded, and deterministic processing still works if Gemini is unavailable.
  • Firebase Authentication tokens are exchanged for revocable, secure, HTTP-only sessions. The hosted environment is email-allowlisted and secrets remain server-side.
  • The hosted reference uses Firebase Hosting, Cloud Run at min zero/max one, Firestore for workflow state and small artifacts, Secret Manager, Direct VPC egress, and an internal-only DataHub Core VM that starts on demand and stops after 30 idle minutes.
  • The web app is request-driven: no timers, no Firestore listener, and no background status polling. An idle tab makes no application requests.

Challenges we ran into

Making DataHub load-bearing without turning it into a lookup widget

Reading a dataset name is easy. Making ownership, classification, glossary meaning, and field lineage change whether a contract can compile is harder. We built a canonical context snapshot, capability discovery, normalization for real MCP response shapes, and honest failure states for missing context.

Separating AI explanation from proof

The strongest-looking shortcut was to let the model summarize a dataset and call it safe. We instead made every proof state a state-machine consequence of deterministic evidence. Gemini can inspect only normalized context, contracts, aggregate evidence, and policy-compatible plans; it cannot query raw rows, run arbitrary SQL, approve, publish, or mutate DataHub.

Proving that remediation preserved utility

Removing every suspicious field can produce a “safe” but useless training set. LATCH executes each alternative before displaying metrics, enforces required fields and a row-retention floor, and re-runs utility checks on the separate candidate.

Keeping an honest hosted demo inexpensive

DataHub Core needs more memory than a free micro VM. We placed it on an internal-only on-demand VM with persistent catalog state and a 30-minute idle shutdown, while Cloud Run scales to zero. We also removed an early polling design after it caused Firestore read amplification; status is now fetched only on page load, explicit action, or manual refresh.

Accomplishments that we're proud of

  • A real DataHub → BigQuery block → measured alternatives → approval → materialization → re-verification → publication → passport → verified DataHub writeback path.
  • DataHub is required both before execution and after publication: it shapes the contract and inherits the governed result.
  • Raw customer conversations remain in BigQuery; LATCH and Gemini persist only bounded aggregates, hashes, job references, policy evidence, and approved artifacts.
  • Multiple remediation plans are evaluated before the user chooses, rather than showing guessed scores.
  • Proof states cannot be assigned by the frontend or Gemini.
  • The Release Passport makes the exact request, context, policy, evidence, SQL, approvals, and output independently inspectable.
  • The hosted architecture has exact-origin security, HTTP-only sessions, CSRF protection, owner scoping, rate limits, BigQuery allowlists, bounded agent steps, private DataHub networking, and server-side secrets.
  • Source tests cover policy compilation, state transitions, proof gates, query safety, DataHub MCP normalization, Firestore query behavior, passport hashing, agent schemas, and the no-polling invariant.

What we learned

  1. Metadata is most valuable when it changes an action. A lineage graph becomes operational when missing lineage blocks the release or a classification causes a deterministic field check.
  2. AI explanation and evidence must be separate products. A model can make evidence understandable without being trusted to manufacture it.
  3. Safety and utility have to be measured together. A release that removes every useful row is not a successful remediation.
  4. Publication is not the end. Writing the governed result back to DataHub gives the next person or agent inherited context instead of another disconnected table.
  5. Cost controls are part of product correctness. Scale-to-zero is not enough if a browser polls every few seconds; request shape and read amplification matter.

What's next for Latch

  • Package customer-owned-cloud deployment into a guided, isolated installer while keeping secrets in each customer's own Secret Manager.
  • Add more versioned release policies and detector adapters for risks that cannot be expressed responsibly with deterministic warehouse checks.
  • Add richer independent-review and separation-of-duties configuration for larger organizations.
  • Support GCS-backed passport retention/versioning as a first-class hosted option.
  • Add expiry-driven revalidation and revocation workflows linked to the DataHub output asset.
  • Contribute reusable DataHub documentation or integration components upstream after the hackathon; this submission does not claim the open-source contribution bonus today.

Built With

Built With — Devpost Tags

DataHub · DataHub MCP Server · BigQuery · Gemini · Python · FastAPI · Next.js · React · TypeScript · Firebase · Firestore · Cloud Run · Google Cloud · Docker · PostgreSQL

Built With — Detailed Stack

Layer Technologies actually used
Frontend Next.js 16, React 19, TypeScript, Firebase Web SDK, Lucide, Manrope, IBM Plex Mono
Backend Python 3.11+, FastAPI, Pydantic, SQLAlchemy, Uvicorn
AI Google Gemini API via the Google Gen AI SDK; configurable Gemini Flash-Lite model; typed bounded orchestration
DataHub DataHub OSS / Core Platform, official DataHub MCP Server, DataHub Python SDK, REST emitter and graph client, schema/ownership/tags/glossary/table-lineage/field-lineage reads, verified writeback
Warehouse/data Google BigQuery and the BigQuery Python SDK; deterministic Standard SQL checks and transformations
Database PostgreSQL for local/self-hosted operation; Cloud Firestore for hosted workflow state and small artifacts
Authentication Firebase Authentication and Firebase Admin; secure HTTP-only session exchange
Deployment Firebase Hosting, Cloud Run, Compute Engine, Direct VPC egress, Secret Manager, Artifact Registry, Cloud Build
Storage Local artifact adapter, hosted Firestore artifact adapter, optional GCS adapter
Testing/CI Pytest, pytest-asyncio, Ruff, mypy, Vitest, ESLint, TypeScript, GitHub Actions

DataHub Technologies Used

DataHub Technologies — SELECT THESE

Technology Why we can honestly select it How LATCH uses it
DataHub OSS / Core Platform The reference runtime operates a real DataHub Core catalog, and the workflow depends on its context graph. Stores and serves source/output assets, schema, ownership, tags, glossary terms, table lineage, field lineage, and release writeback.
DataHub MCP Server LATCH uses the official MCP server at runtime, discovers its tools, maps required capabilities, and performs real context reads. Retrieves entity, schema, and lineage context used to compile the release contract; missing capabilities stop or limit the workflow honestly.
Other The form has no separate SDK checkbox, but LATCH genuinely uses the DataHub Python SDK and REST/graph clients. Seeds synthetic-source metadata, writes output properties and table/field lineage, then reads the aspects back and verifies them. Enter: DataHub Python SDK + verified REST writeback.

Additional DataHub capabilities used in the implementation: lineage, ownership, tags, glossary terms, schema, and writeback.

Built With

Share this project:

Updates