-
-
Signal Evidence Graph
Inspiration
Agents write confident evaluations and give you no way to check what they weighed. Worse, they read missing evidence as refutation: nothing upstream of a claim comes back as "unsupported" rather than "we cannot tell from here," and those are completely different statements about a project.
We wanted a system where "not enough is available to judge this" is a first-class outcome, and where every conclusion traces back to specific artifacts. That turned out to be a context problem rather than a prompting problem, and DataHub already had the pieces: canonical entities, typed lineage, glossary terms, tags and assertions. So DataHub became the system of record instead of something we reimplemented behind it.
What it does
Project materials support claims, claims support an evaluation, and each of those relationships is a typed edge in DataHub. The diagram is one real run: five project materials (the demo recording, the README, the source, the tests, the platform files) supporting seven claims about Save Pip, feeding one evaluation of project depth. Six claims end up substantiated. One, "Publicly accessible on Reddit," stays unverified.
The demo answers a single question, and answers it twice, because the answer depends on what is in the graph:
Does the available project evidence substantiate that Save Pip is more than a polished three-rescue prototype?
In the video-only context DataHub holds the 59-second recording and the frame analysis derived from it. Two claims have nothing upstream at all, and cannot: the recording shows three rescues and then replays the same round, so nothing in it speaks to how much game exists. Those claims render with a dashed border and "no evidence in this context." Unsettled, never false. The agent runs and says so.
Then the context expands. Eleven more artifacts get written into DataHub, each carrying its evidence role, and every previous finding is invalidated at the same moment because none of them weighed this evidence. The agent runs again on the same question with the same prompt.
| Video only | Full project packet | |
|---|---|---|
| Artifacts in DataHub | 2 | 13 |
| Typed lineage edges | 18 | 47 |
| Claims with no upstream artifact | 2 | 0 |
| Claims the evidence still cannot settle | 3 | 1 |
Three of seven substantiated becomes six of seven. The evaluation changed because the available context changed, not because a different prompt was used.
How we built it
Two clients, one store. Neither client keeps a copy of anything.
A Signal Packet plus a sibling overlay file (contexts and evidence roles) is mapped by model.py into DataHub entities: datasets on the signalpacket platform with subtypes Material, Analysis, Claim and Conclusion, rubric criteria as glossary terms, and typed UpstreamLineage edges carrying properties.role. The roles are the substance of the graph: demonstrates, declares, implements, verifies, deploys, analyzes, informs. A README that only declares something is the weakest support in the graph; a passing test that verifies it is the strongest, and the agent is told to weigh them that way.
The Evidence Map reads the graph over HTTP. graph_api.py gets entity metadata, tags, terms and assertion run events over GraphQL, and gets edges plus their roles from the UpstreamLineage aspect directly. The frontend is vanilla HTML, CSS and JS with no build step: cards in a CSS grid, and an SVG layer drawing role-coloured connectors from the resulting DOM rects.
The agent reads the same graph over MCP and nothing else. It has no database credentials and no local copy. It sees the world through mcp-server-datahub over stdio (search, get_entities, get_lineage), weighs the evidence roles it finds, and writes its findings back as custom DataHub assertions with run events. run_agent() takes an optional event sink, so the server observes the existing agent loop rather than reimplementing it, and unknown tools degrade to a generic label so the activity strip never shows work that did not happen.
Nothing in web/ decides an outcome. Findings are re-read from DataHub before the page renders them. Empty DataHub, empty page.
Challenges we ran into
Typed lineage has no first-class home for a role. The SDK's add_lineage helper has nowhere to put one, so edges are emitted as raw UpstreamLineage aspects with properties.role. Reading them back the same way turned out to be a bonus: aspect reads are strongly consistent, so an expanded context renders immediately instead of after the search index catches up.
The two clients see different consistency. The agent traverses lineage through MCP, which does go through the search index. So wait_for_lineage blocks after an expansion until the index agrees. We wait on DataHub rather than inventing edges to paper over the lag, and if the wait times out the CLI and the activity strip say so instead of letting the agent reason over a half-built graph.
Old verdicts describing a graph that no longer exists. A verdict is a statement about the artifacts that were upstream when it was made. The moment eleven more arrive it stops describing this graph. Expanding therefore does two things at once: ingests the new artifacts and invalidates the previous run. But invalidating is not deleting. The verdict:* triage tags come off while every assertion run event stays in DataHub, and the read model serves only the finding whose recorded context matches the context currently loaded.
A claim nothing in the packet can settle. Public reachability is not a matter of weighing evidence roles. The Devvit manifest, the server source, the portal listing and the recording all describe the app without ever opening it, and r/SavePip could not be checked independently because Reddit returned HTTP 403 to unauthenticated requests. So the claim declares the check that would settle it as a property of itself (substantiation_requires, substantiation_bar), artifacts declare what they provides, and the single write path caps the recorded verdict at insufficient-context when no upstream artifact performs the named check. What the agent asked for is kept in the run event as verdict_as_written, so the adjustment is auditable in DataHub rather than silent. This is a property of the graph, not a special case for one claim id; grant the portal artifact "provides": ["verified-public-url"] and the same write goes straight through.
Not forking the shared schema. The vendored Signal Packet schema sets additionalProperties: false, and it is shared identically by all three projects. Evidence roles and contexts are things only this project needs, so they live in a sibling overlay file. data/save-pip.json stays a plain valid Signal Packet, and packets without an overlay still map exactly as they did before roles existed.
Accomplishments that we're proud of
One assertion URN with two run events, FAILURE (insufficient-context, video only) then SUCCESS (substantiated, full packet). Not two unrelated assertions: the same one, changing its mind as the evidence arrived, visible in DataHub's own Quality tab. It is the single best artifact in the demo.
Every claim traced to something actually read or watched, before any of this was built. The evidence inventory is committed, each record carries a supported scope and a limitation, and a test asserts every published claim cites a record. Findings that would have made the demo more impressive were deliberately excluded, and the inventory says why. The numbers were measured rather than read off a README: the test suite was executed (7 files, 695 passing) and the content pool was counted by importing the real modules (6 bespoke plus 100 catalog, so 106).
Tests that assert the properties the demo would be dishonest without. That two claims have zero upstream artifacts in the video-only context and both gain them on expansion. That expanding only ever adds. That roles match the kind of artifact. That a finding reached in one context is never served as a finding about another. That the deployment claim cannot be written back as substantiated while nothing performs its check. That the conclusion entity's name carries no outcome, so it reads the same in both contexts.
Independent verification by design. Every card in the Evidence Map deep-links into the native DataHub UI, so a judge can check the entities, the lineage, the properties and the assertion history without trusting our frontend at all.
What we learned
Typed edges carry most of the value. "What supports this claim" is useful; "how does it support it," declares versus implements versus verifies, is what makes weighing possible and what makes a README-only claim visibly weaker than a tested one.
Insufficient context is a finding, and enforcing it in the write path is far more reliable than asking a model to be humble. The write path is one function, used by the agent and the CLI alike, which is what made the guarantee cheap to state and easy to test.
Giving the agent no credentials and only an MCP read path kept everything honest. The agent's "insufficient context" is a true statement about the graph because there was no other way for it to see anything, and the activity strip is a real trace rather than a progress animation.
DataHub's assertion history is the right home for a changing judgement. We started out planning to version verdicts ourselves and then found the platform already models exactly this.
What's next for Signal Evidence Graph
Ingesting packets directly from the two sibling projects, Signal Packet API and Signal Review Reel, so written evidence and video evidence land in one lineage and the graph covers a full submission rather than a curated one.
More roles, and multi-submission comparison: once evaluations are graph queries, "which of these forty submissions rest only on declares edges" becomes answerable.
Upstreaming what belongs upstream. A role-aware lineage helper would save anyone else emitting raw aspects for the same reason, and the assertion write-back pattern is worth an RFC.
Log in or sign up for Devpost to join the conversation.