Provenance

Inspiration

As an undergrad researcher, I cross-checked three passive-voice detection libraries. They disagreed far more than I expected. My professor put it plainly: anyone can train a model; the hard part is a careful ground truth. Most labeling tools treat agreement as accuracy, and they skip the license, the source, and how wide the uncertainty is. Provenance records those for every kept item.

What it does

Provenance is local. The command is gt. With no arguments it opens http://127.0.0.1:8765/ . The page binds to 127.0.0.1. It is not hosted, and it does not take payment.

Chat is the front door. You type a sentence, such as "Classify these reviews as positive, negative, or mixed," upload up to 20 files (.txt, .md, .jsonl, .jpg, .png, .webp), pick a license you have (cc0, pdm, by, by-sa), and enter a contact email. That email is only the user agent for image search. Text must be uploaded. Images can be uploaded, or, with no files, searched on Openverse and Wikimedia Commons only: 1 request per second, at most 24 images, licenses limited to cc0, pdm, by, and by-sa. NC and ND stay out. Faces are excluded. A prompt with an arbitrary URL, or that asks to skip licenses or collect emails, phone numbers, or faces, is refused.

Two labelers vote. For text, one is word overlap with the class definitions. The other is a local Ollama model when one is running, otherwise a word list from the class definitions. A local LLM is a voter. It is not the model you export. The run then stops for a person to label a random audit, with those votes hidden. Those labels are the ground truth.

After each saved label, once two classes each have two labels, a small model refits on your labels only. Leave-one-out is stored. Text fits are naive Bayes. Image fits are color centroids. CLIP, Ollama, and the word list are not retrained. Models lists every classifier in the workspace and downloads model.json. Score new text at http://127.0.0.1:8765/app or with the SDK. The scored text is not stored.

Where it came from is the record under the chat: dataset/manifest.jsonl with label_source of human_audit, human_queue, or machine_agreed, plus the license, the creator, and each labeler's vote. Human-verified rows are only labels a person entered. Machine-agreed rows carry the 95% Wilson interval and n. Agreement between labelers is not accuracy. A small audit is a wide interval. The data card is not legal advice.

from gt.sdk import Manifest, Model
model = Model.load("gt-work/studio/export/<run>/model.json")
print(model.score("I love this blender. It is sturdy and excellent.").as_dict())
rows = Manifest.load("gt-work/studio/export/<run>/dataset/manifest.jsonl")

How we built it

Python, Typer, one SQLite file, FastAPI on localhost, and a keyboard audit. The chat, the provenance view, and the model list share that database. Licenses fail closed. A PII regex excludes email, phone, and SSN-like text from training. Image search is rate limited. Image labelers, when installed, are CLIP ViT-B-32 and local Ollama (moondream).

Challenges we ran into

A sentence can start a run, but it cannot invent the ground truth. The audit has to be a person, with the labelers' answers hidden. Two labelers can agree and still be wrong, so a small sample stays a wide interval. The exported model has to wait for those labels. A local LLM can vote, and it is not the file you download.

Accomplishments that we're proud of

One sentence and a small upload produce a model you can download, plus a manifest that says where each kept label came from. The chat is the task. The chart is the proof. The page never leaves the machine.

What we learned

Training a model is the easy part. The labels are the value, and they count when a person is the ground truth. A fluent model, local or hosted, is another vote. A small sample is a wide interval. Agreement between labelers is not accuracy.

What's next for Provenance

A hosted workspace as a subscription, with more than one model library on one account. This build stays local and does not charge. A larger audit, so the interval on machine-agreed rows can narrow. Text stays something a person uploads. Licenses stay fail-closed.

Built With

• ai