DOPPEL

DOPPEL turns sensitive production datasets into privacy-safe synthetic twins that preserve useful structure and relationships, using DataHub context for governance, lineage, and verification.

Comment

Challenge fit

Primary — Agents That Do Real Work. DOPPEL ships an LLM agent (python -m app.agent --asset <id>) that reads DataHub context, reasons over every column to classify it and author a governance analysis, hands the plan to a deterministic engine that generates and fail-closed verifies the twin, then writes back the results plus a natural-language knowledge handoff to the catalog (UpstreamLineage, InstitutionalMemory) so the next person or agent inherits the reasoning. Crucially, the model only proposes classifications (validated against the semantic-type enum) and writes the handoff — it can never override a privacy or integrity gate.

Also — Open / Wildcard. DOPPEL uses DataHub as the governance foundation and contributes a reusable DataHub Skill (skills/create-synthetic-data-product/) so any compatible agent can run the same read → reason → generate → verify → write-back loop.

Inspiration

Product teams, contractors, and agents constantly need realistic data to build and test against. The source data usually contains direct identifiers, protected health information, financial records, or other restricted attributes. Handing over production rows exposes the organization; handing over naive fake rows breaks application logic because keys, distributions, correlations, and relationships no longer match.

We wanted a tool that treats the data catalog — DataHub — as the source of truth for governance, so the synthetic twin inherits the same schemas, tags, owners, domains, and lineage as the source. If the catalog says a column is PII, the twin must never expose it. If the catalog declares a foreign key, the twin must honor it. If the catalog records an owner, the twin must carry that owner forward.

What it does

DOPPEL is a metadata-aware synthetic data product generator, driven either from a five-screen UI or by an autonomous agent.

  1. Reads catalog context — schema, primary/foreign keys, tags, glossary terms, ownership, domains, and lineage from DataHub (or a checked-in fixture).
  2. Reasons (agent mode) — an LLM reviews every column, classifies ambiguous ones, flags direct and quasi identifiers, and writes a governance handoff. The deterministic gates remain the sole authority on safety.
  3. Plans generation — assigns a typed strategy to every column based on semantic type and governance tags.
  4. Generates linked tables — parent tables first, surrogate identifiers, distribution-preserving dates, categorical and numeric modelling, and relationship-preserving foreign keys.
  5. Verifies fail-closed — exact-row overlap, direct-identifier overlap, singling-out risk, distribution similarity, correlations, conditional relationships, aggregate-query similarity, and referential integrity.
  6. Writes back to DataHub — registers the synthetic datasets with SYNTHETIC and NON_PRODUCTION tags, source lineage, owner/domain, scores, timestamps, expiry, and a linked evidence report plus the agent's knowledge handoff.

The healthcare demo turns ~1,200 patients and ~4,200 encounters into a VERIFIED development twin with zero copied rows, zero leaked identifiers, and ~98.7% utility. The app also ships finance and retail demo assets.

How we built it

  • Backend: FastAPI + Pydantic, pandas/numpy/scipy for generation and verification, Faker for direct identifiers, acryl-datahub for catalog read/write.
  • Agent: an LLM planner (OpenAI-compatible gateway) that reads the catalog, reviews the per-column classification, and authors the governance handoff — then delegates to the deterministic engine. Falls back to deterministic classification if no LLM is configured.
  • Frontend: Plain HTML/CSS/JS dashboard that consumes the SSE /api/runs/stream endpoint so progress reflects real backend stages.
  • Generation engine: Deterministic, seeded multi-table synthesizer with surrogate IDs, empirical quantile sampling, Gaussian copula for correlated numerics, and FK-safe parent-first generation.
  • Verification: Lightweight deterministic metrics — KS/TVD for distributions, Pearson correlation similarity, grouped-mean preservation, joint-distribution relationships, and cardinality shape comparison.
  • DataHub integration: Reads SchemaMetadata, GlobalTags, GlossaryTerms, Ownership, Domains, and UpstreamLineage; writes DatasetProperties, GlobalTags, UpstreamLineage, SchemaMetadata, and InstitutionalMemory evidence.
  • DataHub Skill: A reusable, upstream-ready skill contribution in skills/create-synthetic-data-product/ that teaches any compatible agent the same fail-closed workflow.
  • Deployment: Live demo on Vercel (serverless Python) at https://doppel-datahub.vercel.app.

Challenges

  • Real catalog schema is messy. Field type classes, tag/term URNs, and foreign-key references differ between fixture data and a live DataHub GMS. We had to make enrichment tolerant of missing aspects while still using the catalog when it is present.
  • Keeping the agent honest. The LLM assists with interpretation and the knowledge handoff, but must never be able to talk its way past a failed privacy or integrity check. We constrained it to propose classifications (validated against an enum) and left the fail-closed gates as the sole authority.
  • Determinism vs. realism. Jittering dates and adding noise makes data realistic, but using wall-clock time for clamping or age calculations would break reproducibility. We fixed reference dates to source-derived values so the same seed always produces the same output.
  • DataHub writeback idempotency. Re-running the pipeline must not create duplicate datasets or lineage edges. DataHub aspect overwrites make this work, but we still validate it in integration tests.

Accomplishments that we're proud of

  • An agent that does real work: reads the catalog, reasons with an LLM, acts through a deterministic engine, and writes reasoning back for the next agent.
  • End-to-end DataHub path: read metadata, generate, verify, publish lineage/tags/scores, and attach evidence.
  • Deterministic output verified: same seed + same input yields identical rows; different seeds yield disjoint identifiers.
  • Privacy gates proven: zero exact-row overlap and zero direct-identifier overlap on the full healthcare dataset.
  • Referential integrity proven: all encounter patient_id values resolve to a generated patient.
  • Reusable DataHub Skill prepared for upstream contribution.

What we learned

  • Metadata-first generation is far more robust than hand-coded column lists. When DataHub is the source of truth, changing a tag or adding a column automatically changes the twin.
  • The right division of labor for an agent is LLM-for-interpretation, deterministic-for-safety. Judges and reviewers need to see the proof, not just trust the model.
  • Synthetic data tools must be honest about what they guarantee. We explicitly label the checks as heuristics and tag outputs NON_PRODUCTION.
  • DataHub's aspect model makes idempotent writeback straightforward once you treat every write as an upsert.

What's next for DOPPEL

  • Read DataHub through the MCP Server / Agent Context Kit so the agent uses first-class DataHub agent tooling.
  • Optional differential-privacy or k-anonymity modes for stricter guarantees.
  • Connectors beyond CSV, starting with SQLAlchemy/PostgreSQL and BigQuery.
  • Row-level access control and audit logging for artifact downloads.
  • Multi-agent review workflow separating generation, verification, and approval.

Built With

Share this project:

Updates