• 

  Home Page or User Query Page

• 

  Loading Page & Agentic Workflow Status!!

• 

  Different Stages in the entire Agentic workflow

• 

  Scoring Part in the validation loop

• 

  One of the artifacts generated

• 

  GitHub Pull Request Modal

MetaLoom AI — Metadata-Aware Code Generation & Development

Inspiration

Every data team has the same painful moment: someone asks for "a validation query on the billing table" or "a DAG for the healthcare pipeline," and an LLM confidently produces something that looks perfect and references a column that doesn't exist. The code compiles. The SQL is elegant. The table name is a hallucination.

The metadata to prevent this already exists — it's sitting in DataHub, catalogued and lineage-mapped. It just never reaches the model. So we built the missing wire: a pipeline where a read-only discovery agent reads your real catalog first, and every downstream artifact is grounded in that handoff. If the request can't be grounded in the catalog, the pipeline stops rather than inventing a schema.

The second inspiration was trust. "The AI says it's good" is worthless. We wanted a number a reviewer could argue with — a score computed in Python from evidence, not a vibe from a model grading its own homework.

What it does

You type a plain-English request. MetaLoom AI returns validated, production-ready artifacts — such as dbt models, Airflow DAGs, Prefect flows, Dagster jobs, SQL scripts, ingestion configurations, migration scripts, documentation, helper utilities, and architecture diagrams. Every generated artifact is then validated and repaired automatically before it can be published, ensuring accuracy, traceability, and production readiness— and opens a pull request against your GitHub repository.

user query
    ↓
[0] preflight    DataHub up? Azure configured? git? graphviz?
[1] discovery    Agent 1 — read-only DataHub retrieval → discovery_context.md
[2] codegen      Agent 2 — writes real files into a sandboxed run directory
[3] validation   Agent 3 — LangGraph loop: collect → judge → guard → repair
[4] publish      push branch + open PR
    ↓
summary + pipeline_report.json

Agent 1 — Discovery. Wired with the DataHub Agent Context Kit's LangChain tools, include_mutations=False. It searches the catalog, pulls entity URNs, schema fields, lineage edges and upstream transformation SQL into a structured handoff with a mandatory Gaps/Unknowns section. A non-negotiable responsible-AI guardrail evaluates before any retrieval: prompt injection, mutation requests, exfiltration and scope escalation emit a blocked marker and the pipeline halts with exit code 2.

Agent 2 — Code generation. Reads only that handoff. Four sandboxed tools — write_artifact, render_dot_diagram, execute_python, list_artifacts. It picks the language, framework and file layout from context rather than a template. Writes are parsed at write time, so broken Python/YAML/JSON is rejected before it lands and the model corrects itself. Every catalog gap becomes an explicit, overridable config input — never buried in business logic.

Agent 3 — Validation & repair. A LangGraph loop that judges, scores, guards and repairs across up to four rounds. Six weighted dimensions, deterministic maths:

Dimension	Weight	Checks
discovery_fidelity	30	Names, URNs, columns, lineage, SQL match discovery exactly
request_coverage	20	Every requirement is in executable code, not a comment
artifact_completeness	15	Files exist, complete, no TODOs or truncation
correctness_executability	15	Parses, compiles, acyclic DAG, parameterized SQL
diagram_code_consistency	10	.dot + .png encode the graph the code actually builds
assumption_hygiene	10	Gaps surface in assumptions.yaml, README, file headers

score = 100 − Σ min(blocker×25 + major×8 per dimension, dimension weight)
            − min(total minor×3, 4)
PASS = score ≥ 90 AND zero blockers

MetaLoom console — a React 18 + Vite Claymorphism operator UI over an SSE-streaming Starlette API. Live stepper, per-stage previews (discovered entities with URNs, lineage threads, generated file browser with hand-rolled syntax highlighting, score breakdown and findings), and a publish dialog. No UI kit, no icon package — every clay surface, icon and animation is hand-built.

How we built it

Layer	Choice	Why
Catalog	DataHub (local GMS via Docker)	Source of truth for entities, schema, lineage
Catalog tools	DataHub Agent Context Kit ([langchain])	Ready-made LangChain tools; mutations filtered out
Agents	LangChain create_agent + tools	Tool-calling loops for discovery and codegen
Orchestration	LangGraph	Both the top-level pipeline and the validate/repair loop
Model	Azure OpenAI via langchain-openai	Per-role deployments — mix model sizes per agent
API	Starlette + Uvicorn	Thin BFF over the same pipeline; SSE for live stages
Diagrams	Graphviz (graphviz + dot CLI)	DOT is generated and rendered, then diffed against the code's real graph
Frontend	React 18 + Vite, plain CSS	Client-rendered localhost console — nothing to server-render
Config	Pydantic-validated Settings	Every env var validated in one place, before any API call

The whole thing is one LangGraph with conditional gates. Every node is wrapped by stage_node, which converts any exception into a recorded error and routes straight to finalize — so the pipeline always writes pipeline_report.json and the operator never sees a raw traceback. ~8,200 lines of Python across 40 modules, plus a 45-file React app and a pytest suite covering scoring maths, path safety, GitHub URL handling and pipeline wiring.

Challenges we ran into

The validation loop wouldn't converge. Our first version oscillated forever: the judge would fix three findings and invent four new stylistic complaints, so the score wandered and never crossed the threshold. Three mechanisms fixed it — scope freeze (round 0 establishes the baseline defect list; later rounds only score baseline findings, deterministic findings, and judge-flagged regressions), guard + snapshots (every round is snapshotted; if a repair lowers the score, the best round is restored from disk and the loop stops), and a minor cap (cosmetic findings cost at most 4 points total, so a 90 threshold stays reachable).

Models grade themselves generously. Asking an LLM "is this good, out of 100?" produces 95 every time. We inverted it: the model returns only evidence-backed findings with a severity and dimension, and Python computes the score. Certain defects — syntax errors, empty files, missing PNGs, stale renders, TODO markers — are found by deterministic checks that never involve the model at all.

"Does the diagram match the code?" is not a question you can ask a model. So we don't. graph_extract.py parses the .dot with a regex edge extractor and unrolls Airflow's a >> b >> [c, d] chains via the Python AST — including chain(), set_upstream/set_downstream, @task decorators and variable→task_id resolution — then hands the judge two edge sets to compare.

Sandboxing a model that writes files. Every tool path resolves against a ContextVar tool session scoped to the run directory. Tools return "REJECTED: ..." strings instead of raising — a raised exception aborts the agent loop, but a returned rejection lets the model read the parser error and fix itself.

Accomplishments that we're proud of

• Grounding that actually holds. The examples in the repo score 97–100/100 against a 90 threshold, converging in 1–2 rounds — on real requests like circuit breakers over healthcare billing, staleness checkers for the taxi pipeline, and FK migration scripts.
• A score you can audit. Open VALIDATION.md in any run: final score, per-dimension breakdown, every finding with its evidence and required fix, the full round history, the repair log, and the stop reason. Nothing is hidden behind "the model said so."
• It refuses to hallucinate. Point it at an empty catalog and Agent 1 correctly blocks rather than inventing tables. That failure mode is a feature we designed for.

What we learned

Determinism is the feature. Every time we moved judgement from the model into Python — parsing, graph extraction, scoring, path validation — the system got both more reliable and easier to explain. The model is best used as a findings generator, not an arbiter.

Convergence needs a frozen exam. An agent that can redefine "correct" each round will never finish. Scope freeze was a bigger unlock than any prompt change.

Reject, don't raise. A returned error string keeps the model in the loop and lets it self-correct; an exception ends the run. Same information, completely different outcome.

Guardrails belong before retrieval, not after. Evaluating the request first — and explicitly carving out that "create X" is the pipeline's normal purpose, not a violation — avoided a false-positive rate that would have made the system unusable.

Metadata quality is the ceiling. The pipeline is exactly as good as the catalog behind it. Discovery's Gaps/Unknowns section turned out to be the most valuable part of the handoff: it's what stops Agent 2 from filling silence with invention.

What's next for MetaLoom AI

Write back to the catalog. Discovery is read-only by design, but the artifacts we generate are metadata. Emit lineage and assertions for generated pipelines back into DataHub so the catalog improves with every run — closing the loop instead of only consuming it.

Execute what we generate. Today correctness_executability means "parses and compiles." The next tier is running generated SQL against a sampled warehouse and generated DAGs in an ephemeral scheduler — turning static confidence into observed behaviour.

Self-hosted and cost controls. Per-role deployments already allow mixing model sizes; next is a local-model path for discovery (cheap, high-volume, tool-heavy) with a frontier model reserved for the judge, plus per-run token budgets surfaced in the console.

Built With

• agent-context-kit
• azure-openai
• css
• datahub
• docker
• github-api
• graphviz-dot
• javascript
• jsx
• langchain
• langgraph
• multi-agent-system
• pydantic
• python
• react
• server-sent-events
• starlette
• uvicorn
• vite