Forget-Me Flow

Compile the forget request—don't guess it. DataHub-grounded remediation planner: lineage → reviewable plan → sandbox SQL → catalog writeback.

Comment

Inspiration

A “forget me” request looks simple until you open the warehouse. Customer identity is scattered across tables, aggregates, BI extracts, and ML features. Catalogs show assets; they rarely hand you an ordered remediation plan with SQL you can review, a human approval gate, and durable evidence the next engineer can inherit.

We were inspired by that gap: not another compliance chatbot, and not an autonomous production eraser—an engineering compile loop grounded in DataHub’s context graph. The pitch we kept returning to: compile the forget request; don’t guess it.

What it does

Forget-Me Flow is a DataHub-grounded technical remediation planner:

  1. Intake a subject (masked in the UI); match via server-side HMAC fingerprint (never returned to the client).
  2. Compile against DataHub lineage, owners, tags, and schema (≥3 downstream assets in the demo graph).
  3. Trust View — impact nodes, confidence, blockers from the context graph (not an LLM inventing deletes).
  4. Approve, then run allow-listed DuckDB SQL with $subject_id binds in a local sandbox only.
  5. Write back (release demo) a public plan.json URL + checksum onto the anchor via Rest.li institutionalMemory, verified with read-after-write.
  6. Short-circuit on rematch so completed work is inherited on this deployment.

Honest DataHub claim: we use DataHub OSS / Core Platform — GMS GraphQL (reads) and Rest.li (writeback). There is no MCP Server, Agent Context Kit, Skills, or Analytics Agent in this repo.

This is not a legal determination tool, compliance certification product, or production delete engine.

How we built it

  • UI: Next.js trust flow (intake → compile → trust → approve → execute → writeback).
  • API: FastAPI case lifecycle, approval gate (execute without approve → HTTP 409), privacy redaction.
  • Adapters: fixture mode for offline/CI; gms mode for live DataHub GMS GraphQL + Rest.li.
  • Planner / codegen: Deterministic Python planner + SQLGlot-validated DuckDB SQL.
  • Sandbox: Local DuckDB with DEMO_SANDBOX + path confinement.
  • Private registry: SQLite HMAC fingerprints for match / short-circuit.
  • Demo graph: Fiction-retail assets emitted into quickstart GMS (make emit-datahub), with search-lag polling on make seed.

Architecture diagrams: assets/System Context.png, assets/Component View.png, assets/Architecture.png.

Repo: https://github.com/nag-gude/forget-me-flow

Challenges we ran into

  • Search-index lag after emit. Fresh quickstart GMS often returned <3 downstream assets for tens of seconds after ingest. We added poll/retry (~60s) in seed so demos and bootstrap don’t flake mid-take.
  • Honest integration surface. Early scaffolding flirted with “MCP mode” naming. We standardized on a real GMS GraphQL + Rest.li client and documented that mcp is only a legacy alias—so judges hear an accurate claim on camera.
  • Privacy vs. artifacts. Keeping raw subject IDs out of DataHub, commits, logs, and API fingerprint fields forced HMAC matching, $subject_id binds, and redacting exception handlers—easy to get wrong under time pressure.
  • Writeback URL shape. Institutional memory must point at a public http(s) plan.json (not file://), which pushed us to committed sample outputs under examples/cases/.

Accomplishments that we're proud of

  • End-to-end read + write against live DataHub GMS (GraphQL context + Rest.li writeback with read-after-write).
  • A human-gated sandbox path that refuses execute without approve.
  • Short-circuit inheritance—rematch returns the prior case instead of starting from zero.
  • Committed, reviewable artifacts judges can open without running the stack (examples/cases/case-release-demo-001/).
  • Clear docs: pitch, workflows, deployment, architecture, DataHub connectivity—and an under-3-minute demo video.

What we learned

  • DataHub’s value for agentic workflows is the context graph + durable writeback, not wrapping the catalog in another chat UI.
  • Search/indexing lag is a real demo risk; treat “just ingested” as eventually consistent.
  • Trust boundaries (what an LLM may not decide) are a product feature, not just a footnote—especially for deletion-shaped work.
  • Sample outputs in-repo are part of submission quality for judges who never clone.

What's next for Forget-Me Flow

Built With

Share this project:

Updates