SkillGraph

Inspiration Everyone is shipping agents. Nobody is cataloging them.

We kept watching the same failure happen: an agent is a prompt with tools, and that prompt depends on tools existing, on other skills existing, on reference files existing. Then one day an upstream renames something and the agent silently breaks. The prompt still says add_tag. The real MCP tool is now add_tags. The agent calls something that does not exist, gets nothing back, and fails for a reason that looks nothing like a docs typo.

Then it clicked. Dependencies are lineage. The agent layer is a data estate, and it has exactly the problems DataHub was built to catch: schema drift, dangling references, orphaned assets, unowned assets. So the project became a simple question. What happens if we put the agent layer in DataHub?

What it does SkillGraph catalogs the agent layer in DataHub the way DataHub catalogs tables. Skills and MCP tools become datasets on the agent-skills and agent-tools platforms. Each ## section of a prompt becomes a schema field. A skill citing a tool becomes upstreamLineage. A section citing a tool becomes fineGrainedLineage, column-level lineage for prose, so a tool rename tells you the exact paragraph to rewrite instead of just the file. Frontmatter becomes custom properties, and audit findings become searchable tags.

Pointed at DataHub's own agent skills repo (datahub-project/datahub-skills), SkillGraph reports 16 provably broken references across 11 skills. Every error matches a bug a third party already filed, so judges can verify the findings by reading the upstream issue rather than trusting our output. Seven tool names have drifted from the MCP server, and nine routes point at skills like /datahub-audit that were never implemented, making the handoff a silent dead end.

The repair loop is made provable. The checker reports N errors with file and line. A patch is applied to a scratch copy of the corpus. The checker re-runs on the patched tree, and the patch is accepted only if the error count drops and no new finding appears. Measured result: errors go from 16 to 10, verified, not claimed.

SkillGraph also ships as an agent skill itself, checked by its own checker, and a static demo page is generated by calling the same functions the CLI calls, so nothing on it can drift from the tool.

How we built it Python 3.11 with the acryl-datahub SDK for emission (MetadataChangeProposalWrapper, DatahubRestEmitter, write_metadata_file), the DataHub MCP Server for read and write back, and DataHub Skills as both the corpus and the delivery format. PyYAML for frontmatter, pytest for 54 tests against the real corpus.

Parsing and checking are pure regex and set logic, deliberately no model in the correctness path. Tool citations are validated against a registry extracted by regex from mcp-server-datahub's source, not from a model's memory of what the tools are called. The only model in the stack is agy with Gemini for the one non-mechanical repair leg, and it is never asked whether something is fixed, only what to try.

The audit deliberately needs no DataHub, no Docker, and no API key, because the diagnosis is the load-bearing part and anyone should be able to reproduce it in one command. Pre-generated outputs live in examples/ so the quality can be judged without running anything.

Challenges we ran into Precision was the hardest part. The first run produced 49 findings and 23 were false positives. env_target and remote_entry_url were template variables in an options table, not tools. /dashboard was an example URL path in a micro-frontend doc, not a skill route. datahub-frontend/conf/*.yaml was a path in a different repo. "what feeds into X" was a phrase quoted to hand off to another skill, not a trigger collision. A checker that cries wolf gets turned off, so we verified every false positive against disk before the code changed and pinned each one with a test.

Dogfooding caught our own bugs. The first self-audit found four findings on our own skill file and exposed two real defects in the checker: the CLI only accepted --repo before the subcommand (not the order anyone types) and the unowned check was hardcoded to fire for every skill, which looks identical to a working check in a report.

The honesty problem was a challenge too. Some corroborating issues were filed by an unrelated contributor months ago, and others by another hackathon participant days before we shipped. We chose to state that distinction plainly in the README, and we deliberately filed no upstream PRs because another participant already had PRs open against these exact bugs. Trustworthy findings are the whole argument, so overstating the corroboration would have undercut the project.

Accomplishments that we're proud of 16 provably broken references found in DataHub's own agent skills repo, every one verified against an upstream issue. Section-level lineage, fineGrainedLineage pointed at prose, so a tool rename yields the exact span to rewrite. A repair loop that is measured rather than claimed: 16 to 10, with zero regressions and a test that fails if the mechanical path silently drops an unfixable finding. 54 tests, all against the real corpus rather than fixtures. Emission verified without a live DataHub by reading the written metadata file back through DataHub's own GenericFileSource with zero failures and zero warnings.

We are proud that the tool ships as an agent skill checked by its own checker, and that the first self-audit immediately caught two real defects in the tool itself. And we are proud of the dogfooding outcome that the tool cannot be flattered: the error count after repair is the entire claim, and inflating it is the one thing that would make it untrustworthy.

What we learned Precision is earned, not assumed. Every rule that narrows a citation exists because the corpus produced a false positive without it, and a checker that cries wolf gets turned off no matter how useful it is in principle.

Correctness should never depend on a model. The audit is trustworthy because it is deterministic, and a reference either resolves or it does not. The registry extracted from server source beats any model's recollection of what tools are called.

Measurement beats judgement in repair loops. Applying a patch to a scratch copy and re-running the checker turns "is it fixed" into a number. We also learned that an honest answer to "is that corroboration really independent" is a feature, not a weakness, and that dogfooding catches bugs no test fixture would.

What's next for SkillGraph Run the audit against more agent skills repos and grow the six checks into a community standard for agent dependency health. Add CI gating so every skills repo gets the audit as a pull request check. Push the emission further with domains, ownership, and glossary terms on skill datasets. Extend the measure-then-claim repair loop to any repo of prompts and tools, not just skills. And keep proving the tool on its own output, because the claim on the tin is the claim you can check.