Inner Life - The catalog's biographer

An agent that reads your DataHub catalog every day, writes what it found in four voices people actually read, and writes it back so the next person inherits it.

Comment

Inspiration

People often say data catalog is boring. Incident reports are boring until they are not. Then you are scrambling.

What I wanted to do was to make it a little more entertaining to read in its own way.

The catalog knows things. It knows that two teams define active_customer differently. It knows a table has not been queried in a year. It knows the owner of a pipeline left the company. Nobody reads it, so nobody knows.

Newspapers solved this a long time ago. You do not read the weather because you want data. You read it because someone wrote it for you. So I built the catalog a newsroom.

What it does

Every day it reads the catalog, ranks what it finds, and writes it up in four registers. A serialized drama for the definitions people disagree about. A weather forecast for the trends. A dream for the tables nobody visits. An obituary for the tables that were removed.

Then it writes the stories back into DataHub. Each one is published as a document, and the dataset it is about gets an institutional memory note pointing at it. Open that table tomorrow and the story is attached. The graph remembers.

Nothing is invented. Under every story sits the report: the subject, the ranker that picked it, the conflict ledger, every claim traced to a fact. Click a phrase in the prose and the fact it came from lights up. A draft that cannot cite its way past the checker is never published.

The catalog in the demo is fictional. I generated it so the whole thing runs offline with no credentials. Six provinces, twenty nine datasets. The pipeline, the rails, the write-back and the site are real. Point it at a live DataHub and nothing changes but the data.

How I built it

One spine, four channels. The spine takes a snapshot, extracts typed facts, ranks them, picks a focus, composes, checks, publishes, writes back. A channel is four knobs and two files: a selector, fact weights, a composer profile, a template. There is no channel specific pipeline code.

It reads DataHub through the Agent Context Kit and raw GraphQL, behind one facade. Channels never touch DataHub types.

The checker is the part I care about. It refuses a draft for unknown fact ids, for citation density, for an uncited selection, for a conflict that never reaches the page, for prose repeated from another story, and for fiction that leaks a real table name. Refused drafts get resampled a few times, then dropped. The issue ships without them and records why.

The fiction never sees a real name. An entity map turns URNs into places and people, per channel, so the same table is a depot in the drama and a river crossing in the forecast.

Challenges I ran into

The citations pointed at the wrong facts.

Fact ids are positional. The extractor sorts facts and numbers them. The pipeline republishes its own documents into the catalog, so the next run sees more documents, the sort order shifts, and every later id moves. The stories were composed against one table and the site was built from another. The drama page claimed a six way dispute over active_customer and cited two obituaries about unrelated tables. It looked fine. It was wrong. I fixed it by composing everything against one table, then stamping every story with a digest of that table so the build refuses a mixed set.

The loop repeated itself. I pointed it at a catalog for seven days running and day two was day one with a new date. Same subject, same headline, same eulogy. No single run could show me that. The fix was not a better prompt. It was an editorial rule per channel: the serial revisits its feud and advances it, the forecast and the night board rotate, and the obituary buries a table once and then says nothing.

The prompt was writing the same sentence every day. Six of seven forecasts opened with "the storm that held X finally broke." I blamed the model. Then I read the profile. It said to open on yesterday and gave an example sentence. The model copied the example, exactly as instructed, seven times.

289 dead links. The catalog page rendered 260 citation chips that anchored to a ledger that page did not have. Clicking did nothing, silently. Every one of these was invisible until I looked at the thing a reader sees.

What I learned

Write the rails before the prose. The interesting output is easy. The output you can trust is the work. A checker that only ever passes is decoration. Mine drops stories, and the report says which and why. That is the feature.

Run a daily thing for seven days before you believe it. Bugs that only appear on day three do not appear in a demo. And read your own prompts before blaming the model.

What's next

Fact ids addressed by content instead of position. A weather fallback so the forecast is never silent on a small catalog. Slides keyed to the place rather than the run, so a returning dream reuses its own art.

Built With

Share this project:

Updates