laminar

Production workspace for data engineering

Comment

Laminar

Production workspace for data engineering.

DataHub is the source of context for an organization's data: what datasets exist, how they are structured, what depends on them, who owns them, and which quality and governance rules apply. Laminar brings that context into the development loop.

It watches DataHub and GitHub, gives engineers a live view of background work, and uses DataHub context to investigate changes, review incoming code, run repository checks, and prepare pull requests.

Inspiration

Data platforms change continuously. A column is removed, a table is deprecated, or a schema evolves. DataHub records the new state and already knows which datasets, models, dashboards, and teams are connected to it. The code using that data lives somewhere else.

That gap creates two common failures.

First, metadata changes before the code catches up. A dbt model continues to select a field that no longer exists, and the problem is discovered through a failed job or broken dashboard.

Second, code changes without checking current metadata. A pull request introduces a field that does not exist, depends on a deprecated dataset, or conflicts with an existing quality rule.

We built Laminar to connect those two systems. DataHub supplies the schema, lineage, ownership, assertions, and governance context. GitHub supplies the code and review boundary. Laminar turns the context between them into visible, testable engineering work.

What it does

Laminar is organized around projects. Each project contains its conversations, skills, background tasks, pull requests, and connections to DataHub and GitHub.

Responds to DataHub changes

Laminar includes a DataHub Actions adapter that listens for dataset deprecations and schema changes. A relevant event creates an assigned task in the workspace and starts an investigation.

The agent reads the current DataHub entity, schema, lineage, and assertions, then checks out the connected repository. It finds the affected code, runs the repository's own commands and tests, inspects the diff, and opens a pull request when a repair is needed.

The task runs in the background. Its reasoning, tool calls, command output, status, and result stream into the workspace and remain available after the run completes.

Reviews incoming code against DataHub

Laminar also listens to signed GitHub App events. When a developer opens or updates a pull request, Laminar checks the proposed revision against live DataHub metadata.

The same context works in the opposite direction:

DataHub changed → inspect the code
Code changed → inspect DataHub

Projects can restrict automation by branch, path, GitHub actor, event type, DataHub scope, and maximum capability. Laminar can observe, publish a report, comment on a pull request, or prepare a repair branch.

Gives engineers direct DataHub workflows

The interactive agent includes five focused skills:

  • DataHub Search finds canonical entities and inspects their metadata.
  • DataHub Lineage traces upstream causes and downstream impact.
  • DataHub Quality inspects assertions and quality evidence.
  • DataHub Enrich proposes descriptions, owners, domains, tags, glossary terms, structured properties, and documents.
  • Data Visualization turns verified schemas and relationships into Mermaid diagrams.

Skills are attached to individual chats and read when relevant. Users can also create and import their own skills.

Writes approved metadata back

Laminar can apply metadata changes through DataHub rather than leaving them as text suggestions. Mutation tools pause and show the exact proposed operations inside the conversation. The user approves or rejects them, and the result of every approved write is streamed back into the chat.

An investigation can therefore improve the metadata that powered it. Updated descriptions, ownership, domains, tags, glossary terms, structured properties, and documents become context for the next engineer or agent.

Works with private DataHub environments

Laminar includes an outbound connector for DataHub instances that are not publicly reachable. It runs beside DataHub and opens an authenticated WebSocket to the hosted Laminar service. DataHub API requests travel through that connection to the local GMS server, while DataHub credentials remain in the local environment.

This gives the hosted agent live access to private schema and lineage without opening an inbound port or exposing GMS to the internet.

Produces reviewable artifacts

Laminar's output lives in Git: commits, diffs, reports, and pull requests.

The included sample outputs show two completed workflows:

  • A generated DATA_LINEAGE.md containing the live schema and multi-domain lineage for Kerni's dim_products model.
  • A PR Guard repair where a human introduced the nonexistent product_tier column. Laminar checked DataHub, removed the invalid reference, compiled and ran the dbt graph, passed all 79 data tests, and opened the repair pull request.

The inputs, diffs, generated document, and verification report are included in the examples/ directory.

How we built it

Laminar combines DataHub's context layer with an agent runtime, repository execution, and a persistent web workspace.

DataHub event                         GitHub event
      │                                    │
      ▼                                    ▼
DataHub Actions                       GitHub App
      └──────────────────┬─────────────────┘
                         ▼
               Trigger and policy layer
                         │
                         ▼
              Google ADK + Gemini
                │              │
                ▼              ▼
      DataHub Agent Context   Repository sandbox
                └──────┬───────┘
                       ▼
              Report, comment, or PR
                       │
                       ▼
              Persistent live workspace

DataHub Agent Context Kit

The Python backend uses datahub-agent-context to build Google ADK-compatible tools from a DataHubClient.

client = DataHubClient(server=server_url, token=token)
tools = build_google_adk_tools(
    client,
    include_mutations=include_mutations,
)

Laminar adds explicit error results, project-level scope enforcement, and confirmation-gated mutations around those tools.

DataHubScopeGuard requires searches to include an authorized project scope and restricts later tool calls to allowed or previously discovered URNs. Mutation functions are wrapped as Google ADK FunctionTool instances with require_confirmation=True.

This gives the agent one consistent DataHub toolset for entity discovery, schema inspection, lineage traversal, assertion review, document search, and metadata enrichment.

DataHub Actions and the local connector

LaminarAction implements DataHub's Actions interface. The datahub-actions.yaml pipeline consumes DataHub's Kafka event stream and sends relevant dataset events to Laminar.

The backend supports EntityChangeEvent_v1 and MetadataChangeLogEvent_v1. It compares current and previous schema aspects, extracts field additions and removals, ignores initial ingestion baselines, and creates deterministic delivery identifiers.

The same Actions process can start the outbound connector. The connector uses a project-bound HMAC token, restricts forwarded methods and headers, relays requests to the local GMS server, and reconnects automatically.

Google ADK and Gemini

Google ADK coordinates the root agent, repository specialist, function tools, confirmation flow, sessions, and streaming events.

Laminar uses:

  • Gemini 3.5 Flash for conversation and event-driven investigation;
  • Gemini 3.1 Pro Preview for repository inspection, code changes, tests, and pull-request preparation;
  • Gemini 3 Flash Preview for lightweight chat titles.

The root agent handles DataHub investigation and ordinary conversation. Work that requires repository execution transfers to a specialist with the exact repository capability configured for the project.

GitHub App and repository execution

The GitHub App provides installation-scoped repository access. Incoming events are signature-verified, delivery IDs prevent duplicate work, and installation tokens are requested with only the permissions required for the selected capability.

Repository work runs through a sandbox interface. It clones the exact revision, exposes bounded file and command tools, streams command output, captures the final diff, and handles commit, push, and pull-request creation.

Application stack

The backend is built with Python 3.11, FastAPI, Google ADK, SQLAlchemy, PostgreSQL, asyncpg, httpx, and WebSockets.

The frontend uses React, TypeScript, Vite, Mermaid, React Markdown, and Lucide. It presents conversations, skills, approvals, pull requests, sandbox activity, integration settings, and background tasks.

PostgreSQL stores projects, chats, skills, integration state, webhook deliveries, agent runs, and individual run events. LISTEN/NOTIFY broadcasts run updates across backend instances, and server-sent events deliver them to the browser without a refresh.

Slack incoming webhooks provide completion, blocked, and failure notifications for background work.

Challenges we ran into

Connecting a hosted agent to local DataHub

DataHub Actions could send an event to the hosted service, but the agent still needed to query the private GMS server for current metadata. We built the outbound connector so the complete investigation could remain event-driven without exposing the local DataHub instance.

Turning an event into a correct code decision

A schema event identifies what changed; it does not prove what the repair should be. A removed and added field may represent a rename, two independent changes, or a metadata correction.

We separated the trigger from the diagnosis. The event selects the project and seed entity. The agent then reads current DataHub context, inspects the repository, and runs its tests before deciding what to publish.

Enforcing authority across two systems

GitHub and DataHub expose different kinds of access. We made both explicit project configuration rather than relying on prompt instructions.

GitHub capability controls which repository tools and token permissions are available. DataHub scopes are enforced around tool execution. Metadata writes pause for confirmation and resume only after a recorded user decision.

Presenting unattended work live

DataHub and GitHub can start work when no browser request is active. We persisted every run event, used PostgreSQL notifications to synchronize backend instances, and added project-level and chat-level event streams.

Assigned tasks now appear automatically and open into the exact reasoning, tools, command output, and result produced by the run.

Accomplishments that we're proud of

A DataHub event becomes a tested pull request

The implemented workflow begins with a native metadata event and ends with a GitHub artifact a data team can review. DataHub schema and lineage determine the investigation and the generated repair.

The same context protects both sides of a change

Laminar handles reactive repair and preventive pull-request review with one DataHub evidence layer. This makes the product useful during both incident response and ordinary development.

Private DataHub environments work with the hosted workspace

The connector keeps GMS private while preserving live access to DataHub's schema, lineage, quality, and governance context.

DataHub is both read and write context

Search, lineage, quality, enrichment, and visualization operate inside the same project. Approved metadata changes return to DataHub and become context for future work.

The generated work is easy to inspect

The Kerni examples contain a generated lineage document and a schema-aware repair backed by dbt compilation, 24 successful model runs, 79 data tests, and source identifier verification.

What we learned

DataHub is more than a retrieval layer

Its value comes from connecting technical identity to organizational context. The same dataset can expose schema, lineage, owners, domains, assertions, usage, and governance through one agent toolset.

Prevention and repair are the same comparison reversed

One direction asks which code became inconsistent after metadata changed. The other asks whether proposed code is consistent with current metadata. Building both produced a broader product without a separate reasoning system.

Events should narrow the investigation

The event supplies a precise starting point. Current DataHub evidence, repository inspection, and executable verification determine the final action.

Agent work needs durable evidence

Reasoning is useful while a task runs. Diffs, reports, tests, diagrams, and pull requests are what make the result inspectable and reusable afterward.

What's next for Laminar

Contract-aware development

PR Guard will expand from schema existence to assertion compatibility, deprecation windows, ownership requirements, structured properties, and domain-specific review policy.

DataHub incident history

Schema events, investigations, pull requests, merges, and metadata enrichment will form one timeline attached to the affected DataHub entities.

Durable workflow execution

Background work will move into queue-backed workers with resumable checkpoints, cancellation, retry policy, and replayable trigger fixtures.

More engineering systems

The repository layer will expand to GitLab, Airflow, Dagster, Prefect, ingestion recipes, data contracts, and migration code while DataHub remains the shared context layer.

Built With

Share this project:

Updates