Tally — workspace.json for DataHub

DataHub shows what depends on a dataset. Tally finds the exact code to change, builds an evidence-backed plan, and refuses to guess where the proof ends.

Comment

Inspiration

The most dangerous integration failure is not always an exception. Sometimes it is a plausible zero.

DataHub can tell an agent which dataset matters, what depends on it, who owns it, and how it connects to the context graph. Before that agent can safely change production data code, it still has to answer a more operational question:

Which repository file actually produces this dataset, and how far can the available evidence be trusted?

That handoff breaks in a common repository layout. DataHub carries the dbt file path relative to the dbt project. Git and repository intelligence use paths relative to the repository root. When the dbt project lives under dbt/, the two systems name the same file differently.

I measured the consequence:

  • Project at the repository root: 5 of 5 models resolved
  • The same project nested under dbt/: 0 of 5 models resolved
  • Process exit code in both cases: 0

Nothing crashed. Zero looked like a valid answer.

I built Tally to make that failure impossible to hide and to show exactly how revision bound repository identity changes an agent's proposed action.

What Tally does

Tally is a context safe change impact agent and cockpit for DataHub. It:

  1. Reads the canonical DataHub dataset URN, schema, ownership, and lineage through the official DataHub MCP Server.
  2. Reads the dbt file path DataHub already carries without reconstructing dataset identity.
  3. Normalizes the dbt project relative path into a repository relative source path.
  4. Joins that path to a pinned workspace.json repository artifact.
  5. Emits a typed, evidence bounded ChangeImpactEvent.
  6. Compares the agent's plan with DataHub context alone versus joined repository identity.
  7. Writes a narrow enrichment back to DataHub and observes the before and after states.
  8. Renders Impact, Change plan, and Receipts in an offline cockpit.

The result is not merely a filename. Tally shows how the dataset resolved to code, which system owns each fact, what was actually observed, and where the proof ends.

The shipped end to end proof

I ran one complete judge inspectable path against DataHub OSS using the official DataHub MCP Server and a pinned public Transfermarkt dbt corpus.

DataHub context was read, not reconstructed

The captured event observed:

  • 8 upstream lineage edges
  • 1 downstream lineage edge
  • successful reads in both directions
  • completeness established against pinned readiness manifests

The dataset identity remains the URN returned by DataHub. Tally does not recreate it from dbt naming conventions.

The dataset resolved to exact, pinned code

Tally joined the DataHub dataset to:

dbt/models/curated/game_events.sql

The source is bound to Transfermarkt revision:

59fa295c51fc23466f3a71542f8bf3d1335daa83

This is the coordinate mismatch the product exists to solve:

  • dbt path: models/curated/game_events.sql
  • repository path: dbt/models/curated/game_events.sql

Joined repository identity consistently changed the plan

I ran Qwen Plus under two controlled conditions with the same task, model, prompt, decoding settings, repository revision, and DataHub snapshot. The context envelope was the only varying input.

Across 10 controlled paired runs:

  • Exact source revision in the plan: 0 of 10 DataHub only, 10 of 10 joined context
  • Distinct normalized step sequences: 5 DataHub only, 1 joined context
  • Refusal present in DataHub only plans: 10 of 10
  • Refusal removed after joining repository identity: 10 of 10
  • Partial or failed pairs: 0

The DataHub only result was not a failure. It correctly refused because the repository relative source location was unknown and could not be guessed. Joined context supplied the exact path and pinned revision, allowing the same task to propose a checkable quality edit.

This is a controlled comparison on one task with one model at temperature 0. It is not a significance test or a claim about models in general.

Inspect HAC-150, including raw outputs and aggregation

Evidence was contributed back to DataHub

Tally wrote the workspacejson_evidence_tier structured property through DataHub's GraphQL mutation surface, then reread the state to produce an observed receipt.

The receipt keeps these outcomes separate:

  • accepted mutation
  • no op
  • refusal
  • omission
  • accepted but not yet observed
  • observed success

Tally writes only evidence it owns. It does not overwrite human authored descriptions, risk scores, or unrelated structured properties.

The complete checksummed live evidence package is available in the HAC-152 evaluation bundle.

Nothing vanishes

Every attempted dbt node resolution is classified as:

  • Kept: resolved to repository evidence
  • Dropped: expected to resolve, but did not
  • Excluded: deliberately outside the requested resolution set

The resolver enforces:

kept + dropped + excluded = total

On the pinned proof corpus, all 28 nodes are accounted for:

  • 8 kept dataset bearing nodes
  • 20 test nodes excluded by policy
  • 0 dropped

An empty collection cannot silently stand in for a failed read, unsupported node, or incomplete result.

Evidence axes stay independent

Tally does not let one successful check launder another claim:

  • A successful lineage read does not imply exhaustive lineage.
  • An exact repository match does not imply behavioral co change evidence.
  • A mutation response does not prove that the intended state became observable.
  • An agent plan is not treated as observed metadata.
  • Missing, failed, not queried, indeterminate, and not exposed by the source are different states.

That makes Tally safe to place before an agent, CI workflow, or developer preparing to change production data code. It either shows where the evidence leads or states why the action should stop.

How I built it

Tally is a public Apache 2.0 TypeScript and Node.js application.

  • React 19, Vite, and Tailwind power the cockpit.
  • Zod validates the runtime evidence contracts.
  • DataHub OSS and the official DataHub MCP Server provide the catalog surface.
  • dbt, DuckDB, and Python provide the public data projects.
  • Qwen Plus performs the controlled plan comparison.
  • Vitest covers contracts, adapters, invariants, fixtures, and view models.
  • Playwright and axe cover the judge route across Chromium, Firefox, and WebKit.

The dependency direction is deliberate:

DataHub context graph + workspace.json repository identity -> Tally change evidence

DataHub supplies the canonical dataset identity, schema, lineage, governance context, and writeback destination. workspace.json supplies revision bound repository paths and artifact provenance. Tally is the DataHub application that joins them.

The neutral workspace.json producer remains independent of DataHub. DataHub MCP reads, native dataset identity, dbt path interpretation, plan comparison, cockpit rendering, and enrichment writeback remain inside Tally.

The ChangeImpactEvent and PlanComparisonArtifact are separate contracts. Observed integration facts live in the event. Agent output lives in the comparison and binds back to exactly one event digest. This prevents an inference from masquerading as observed metadata.

Build window and prior work disclosure

I built Tally alone.

I am also the creator and maintainer of workspace.json, the pre existing open source repository context standard that Tally consumes. The standard and its broader ecosystem existed before this hackathon.

Tally itself was newly created during the submission period. The first commit in workspacejson/datahub-agent is dated July 24, 2026. The DataHub integration, MCP transport, evidence contracts, cockpit, evaluations, writeback path, and judging materials were built for this hackathon.

During the same period, I strengthened the public workspace.json CLI and standard around reproducible generation, adapter parity, stable read paths, and contract correctness because building Tally exposed where the producer and consumer boundary needed more rigor. Those ecosystem improvements support Tally, but I do not count them as newly created DataHub application code.

Tally is not a request for DataHub to adopt a proprietary dependency. It demonstrates that revision bound repository identity is a missing input for DataHub agents, and that an open artifact is one reproducible way to supply it.

Challenges I ran into

Three coordinate systems looked identical until they were not

A dbt manifest records original_file_path relative to its project. Git reports paths relative to the repository root. A tool may run from either location.

At the repository root, simple equality appeared correct. In a nested project, it collapsed from 5 of 5 to 0 of 5.

Correcting normalization was only half the fix. The output contract also had to account for every input so a future mismatch could not disappear as another empty list.

The graph converges asynchronously

During a clean live run, an upstream lineage read arrived before DataHub's graph index had converged. Tally refused to reinterpret the unreadable response as zero upstreams.

After convergence, the same official MCP path observed the expected lineage. I captured the timing condition and built bounded readiness evidence instead of hiding it behind an arbitrary fixed sleep.

Proving a write is harder than sending one

A successful mutation response does not prove the intended graph state became visible. Tally reads before and after, polls to a bound, distinguishes success from no op and refusal, and emits a receipt even when the intended state is not reached.

A verifier can look green without doing the work

One early comparison gate could remain green when the official MCP binary was unavailable because no mismatch had been observed. It now reports the comparison as inconclusive and names what did not run. No observation is no longer treated as successful verification.

Accomplishments

  • Built a complete DataHub system integration from a new repository during the hackathon window.
  • Implemented the dataset URN to dbt model to repository file to workspace.json join.
  • Reproduced the silent zero through a controlled 5 of 5 versus 0 of 5 perturbation.
  • Read live DataHub context and lineage through the official MCP server.
  • Ran 10 controlled paired plan comparisons with complete raw outputs and checksums.
  • Wrote a narrow structured property back to DataHub and verified the intended state through reread.
  • Preserved provenance, read status, completeness, resolution, unavailability, evidence tier, and writeback as independent axes.
  • Completed 13 focused correctness and readiness audits.
  • Passed the required TypeScript, lint, clean room, test, browser, production build, and asset governance gates.
  • Shipped a live cockpit, visual README, judging guide, reproducible evaluation bundles, and public source.

DataHub feedback and upstream contributions

Building against a real DataHub instance produced eleven findings and one open question. I excluded one initially suspected DataHub defect after proving that the fault belonged to my own adapter. That correction remains in the feedback record because a trustworthy audit should show where it was wrong.

Read the complete DataHub feedback record

I also opened two upstream contributions generated directly by the integration work:

Both contributions remain open. I do not claim acceptance or endorsement.

What I learned

The most dangerous answer in an integration is often not an exception. It is a believable result produced from incomplete evidence.

Metadata aware development crosses catalog identities, transformation graph identities, project relative paths, repository relative paths, code intelligence keys, and agent plans. A production quality agent cannot merely connect those systems. It must account for every attempted connection, preserve ownership of each claim, and explain where the chain stops.

Observability belongs in the contract, not in debugging output added after a failure. The accounting invariant, digest binding, explicit unavailable states, and observed writeback receipt are what make this workflow trustworthy enough to place before an automated code action.

An agent should not merely know more. It should know and show when it does not know enough.

Limits

  • HAC-150 is one controlled task with one model, not a general model evaluation.
  • Transfermarkt is the judge facing nested project demonstration. Jaffle Shop remains the clean install and proof corpus because its root level layout cannot reproduce the prefix mismatch.
  • The bound event does not establish behavioral co change partners, so Tally does not assert them.
  • The official MCP server does not expose Dataset.externalUrl, so Tally records that gap rather than fabricating a source link.
  • Evidence contract 1.4 now separates declared completeness parameters from the provenance of the read that actually executed. Historical 1.3 evidence remains immutable rather than being rewritten to look newer than what was captured.

What's next

The next step is to validate the same repository identity boundary with another DataHub workflow and explore a first class integration that lets DataHub agents consume revision bound repository context without coupling DataHub to one proprietary tool.

The destination is not another agent that says, "this dataset is upstream."

It is an agent that identifies the responsible code, shows how it reached that conclusion, reveals how the evidence changes the plan, contributes useful context back to DataHub, and refuses to guess when the proof runs out.

Built With

Share this project:

Updates

posted an update —

August 7 — the demo is cut, and the same discipline caught four more silent failures

Two weeks in, the submission is close and the demo is assembled. In the spirit of the earlier updates, here is the honest close of week two rather than a highlight reel.

Cutting the demo turned out to be its own audit. While assembling it I found the same failure class four separate times inside my own tooling: a shallow clone that returned confident, well formed, wrong numbers instead of reporting insufficient history; two scenes that recorded the wrong screen while every content assertion passed; a captured frame that was blank yet passed every check, because the checks read the DOM and the DOM was correct. Each one was a plausible result at the wrong referent. That is the exact failure Tally exists to catch, and the tooling caught them because it was built to look at the output rather than trust the status. Not a comfortable pass, but the clearest possible evidence that the discipline holds when it is turned on itself.

What is solid: the evidence is public, checksummed, and replayable from a pinned commit. Every number in the demo can be verified rather than taken on my word, and the whole thing runs in a committed offline mode with no live network read.

What I will state plainly, because the project is about not overclaiming: this remains one repository, one point in time, one paired comparison at temperature zero. It is a sharply argued, fully reproducible single case, honestly bounded. That is what two weeks of solo work buys, and I would rather tell you that than dress it up.

I built this alone, and at this point the scaffolding is done and the missing piece is people. If the idea of making the handoff between agents and real codebases more honest resonates with you, I would genuinely like to hear from you.

Thanks to the organizers and to the DataHub team for the room to build something real.

Log in or sign up for Devpost to join the conversation.

posted an update —

July 31 — One week ago, Tally was an empty repository.

Today, it is a working, tested system integration that connects DataHub’s metadata graph to repository evidence from workspace.json.

I have been building Tally independently as a solo project. The goal is simple: help an agent move from “this dataset matters” to “this is the exact code responsible,” while making uncertainty and missing evidence impossible to hide.

In the first seven days, the project progressed from its initial scaffold to a complete path that can:

  • Read dataset context and lineage through the official DataHub MCP Server
  • Resolve a DataHub dataset to its exact dbt source file at a pinned Git revision
  • Join that result with workspace.json repository evidence
  • Compare an agent plan with and without the joined context
  • Trace every plan change back to supporting evidence
  • Write a narrowly scoped enrichment back to DataHub
  • Reread the catalog and verify that the intended state became observable
  • Render the full story in a judge-facing React cockpit

The integration has been tested against public, pinned dbt corpora in both root-level and nested-project layouts. The nested case matters because it reproduces the original failure that inspired Tally: a plausible zero caused by two systems describing the same file in different coordinate systems.

The current system passes 669 root tests and 152 cockpit tests. The browser suite exercises the experience across Chromium, Firefox, and WebKit, with accessibility, responsive-layout, clean-room, lint, type, and production-build checks included.

I also ran a 13-part adversarial audit program covering:

  • acceptance criteria and proof traceability
  • contract and invariant enforcement
  • independent lineage reconstruction
  • GraphQL and MCP parity
  • nested-corpus recovery
  • clean-clone reproducibility
  • causal plan differences
  • writeback transaction integrity
  • fault injection and degraded states
  • golden-package provenance
  • cockpit truth rendering
  • cross-repository compatibility
  • DataHub integration-surface coverage

All 13 audit tracks now pass.

The repository currently records 88 commits and 69 merged pull requests, all authored through this solo build effort. Those numbers are less important than what they produced: one public, reproducible path from DataHub context to exact code, an evidence-backed plan, and an observed writeback receipt.

There is still polish and demo work ahead, but the central technical question is no longer hypothetical. The integration works end to end, its claims are tied to inspectable evidence, and its failure states are treated as product behavior rather than hidden exceptions.

That feels like a good first week.

Log in or sign up for Devpost to join the conversation.

posted an update —

July 29 — the project got smaller and more trustworthy

Today’s highest-value progress came from disproving our own implementation assumptions:

  • We removed a dormant URN-reconstruction module after proving its only consumer was its own isolated test. Production already preserves DataHub-returned identity and uses DataHub-carried file metadata.
  • We fixed a verification gate that could report a match when the MCP side never answered. Missing comparisons now produce a distinct INCONCLUSIVE verdict.
  • We captured a moving DataHub catalog, then reproduced both pinned lineage manifests by digest after it changed. That strengthened the topology evidence while falsifying our earlier readiness assumption.
  • We brought ~3,700 lines of evidence-producing scripts under an enforced lint boundary.

The through-line: Tally should never turn “not checked” into “verified,” whether the gap is a missing node, an unanswered MCP call, or a write that was accepted but not observed.

Next: finish the golden cockpit binding, narrow the writeback mutation boundary, and cut the under-three-minute demo.

Log in or sign up for Devpost to join the conversation.