Asterisk

Asterisk checks which subjects your research split actually holds out—with cited rules, source-row evidence, and a replayable receipt.

Comment

Inspiration

Before asking whether a model is accurate, ask what its test actually tested.

Imagine evaluating a medical-imaging model on images it has never seen. The training and test filenames are different. The split appears clean. But some of those new images belong to patients already represented in training.

For a researcher asking whether the model works on new patients, that distinction changes what the evaluation can establish. New images and new people are different claims.

The relevant identity is not always a convenient column. It can be buried in a filename convention, a metadata join or a paragraph of dataset documentation. A script can compare identifiers once they are prepared; someone still has to establish what those identifiers mean.

I built Asterisk to make that assumption visible and checkable.

In pinned public WSI-Bench report-task files, Asterisk measured 0 shared slide identifiers but 42 shared patient identifiers, affecting 43 of 208 evaluation rows, under the documented patient rule. This does not establish the benchmark authors’ intended claim or an effect on accuracy. It demonstrates why the unit being checked must be explicit.

The goal is to turn “our test set is independent” into a specific question, an inspectable rule and evidence someone else can check.

What it does

Asterisk helps researchers inspect whether their train/test split separates the recorded units their claim depends on: images, recordings, patients, specimens or other supported identities.

The workflow has four parts:

  1. State the claim and add the evidence. Specify what must be unseen in training, then add split metadata and the dataset’s documentation.
  2. Preview and propose. Files first reach the Asterisk server. Review the documentation, column summaries and selected identifier examples before sending them to NVIDIA Nemotron 3 Super on Nebius Token Factory.
  3. Review and measure. Nemotron proposes an extraction rule with documentation citations. Inspect or correct it, then let application code measure overlap and expose training/evaluation witness pairs.
  4. Replay and retain. Request a fresh Nebius Sandbox replay, save the result and export a readable receipt or an optional evidence bundle.

Zero overlap does not automatically become “held out.” Complete mapping coverage and confirmation of the identity meaning are required.

The receipt preserves the rule, partition choices, citations, witnesses, model origin and whether the proposal was edited.

How we built it

I built Asterisk with Python, FastAPI, SQLite and vanilla JavaScript.

Nemotron handles the interpretation step: reading the supplied documentation and proposing how a research unit is represented in the files. In the recorded WSI-Bench run, it identified the patient as characters 1–12 of the slide filename and cited the supplied definition.

Its output uses a bounded extraction language supporting operations such as column selection, filename parsing and metadata lookups. Application-owned code validates supported expressions and computes the measurements. Model-written prose cannot supply the overlap count.

A validator can request one repair. A guided editor lets the researcher correct the proposal or define the unit manually.

Nebius Token Factory serves the NVIDIA model; Nebius Sandboxes replays the computation. A fresh isolated Sandbox runs the same engine and inputs, and Asterisk compares its result with the server measurement.

An exported evidence bundle can also be recomputed from a separately trusted Asterisk installation without provider credentials or executing bundled code.

Challenges we ran into

The hardest problem was distinguishing an executable rule from a scientifically meaningful one. A model can produce valid syntax while grouping samples at the wrong level. Structural validation alone cannot settle that question.

The shipping Super prompt scored 5/10 on the hardest retained preparation set, below its precommitted performance threshold. Later candidates failed promotion criteria. Those results remain published, and review, correction and abstention remain part of the workflow.

Provider delays and bounded Sandbox inputs also required explicit failure states and compressed uploads. Failed proposals and failed replays must remain visible rather than being presented as completed checks.

Accomplishments that we're proud of

  • A real benchmark through the actual product. The demonstration takes identifier-only WSI-Bench files through Nemotron, review, measurement, matching Sandbox replay and export. That attempt’s saved provenance records an unedited proposal.
  • Evidence behind every overlap finding. The reader can trace a shared patient identifier to different training and evaluation slide filenames and inspect the rule connecting them.
  • A handover that survives the browser session. The exported HTML opens locally with the measurement, derivations, citations, witnesses and proposal provenance.
  • Visible uncertainty. A separate fictional example demonstrates zero overlap stopping at pending confirmation, with the identity assumption still awaiting review.

Watch the demonstration · Explore the saved public result

What we learned

A correct intersection can still answer the wrong scientific question. The unit of independence belongs in the claim, the computation and the evidence handed to the next reader.

Giving the model a bounded job made its mistakes easier to inspect. Retaining failed evaluations helped distinguish a convincing demonstration from evidence of broader reliability.

A matching replay establishes reproducibility under the recorded rules; it does not authenticate the data or independently validate their scientific meaning. Human confirmation records an assumption. Researcher usefulness, time saved and adoption remain unmeasured.

What's next for Asterisk

The next step is to test whether researchers can understand and use a receipt without its author explaining it.

Preparation improvements need fresh, frozen evaluations before broader reliability claims. Additional dataset conventions should be supported only where their extraction rules and evidence remain inspectable.

The aim is practical: help researchers make the assumptions behind a result explicit enough for another person to check.

Built With

Share this project:

Updates

Submission history