• 

• 

Label Loop

Why we built it

Software can rank uncertain samples, update a classifier, and calculate metrics. A person still needs to decide what an ambiguous example means. Most labeling interfaces expose this process as buttons and tables, leaving a browser agent to infer which actions are safe. We wanted the page to publish those operations directly while keeping label decisions under human control.

Support-ticket routing gave us a small task with visible results. Label Loop classifies tickets as billing, bug, or access.

What it does

Label Loop runs an online multinomial Naive Bayes classifier in the browser. The session begins with nine seed examples, twelve unlabeled review candidates, and a fixed nine-item holdout set.

The agent ranks unlabeled tickets by predictive entropy and can place up to three in the review queue. A person assigns each label through the page. The agent then trains a batch and evaluates the updated model.

The interface shows probabilities, per-class F1, token weights, a confusion matrix, calibration, and log loss. A ledger records label provenance, while a separate trace records WebMCP calls. Model-setting proposals stay pending until a person accepts or rejects them.

How we built it

The project is a static Vite application with no backend or external model API. The classifier and session state run in page memory, so reloading creates a fresh deterministic session.

For a document $d$ and class $c$, the model calculates:

$$ P(c \mid d) \propto P(c)\prod_{w \in d}P(w \mid c) $$

Token likelihoods use Laplace smoothing:

$$ P(w \mid c)=\frac{N_{w,c}+\alpha}{N_c+\alpha |V|} $$

$N_{w,c}$ is the count of token $w$ in class $c$, $N_c$ is the class token total, and $|V|$ is the vocabulary size. Each confirmed example increments these counts.

Review candidates are sorted using normalized predictive entropy:

$$ H(p)=\frac{-\sum_{c=1}^{K}p_c\log p_c}{\log K} $$

A value near one means the probabilities are close, so that ticket moves toward the front of the review queue. One-vs-rest log-likelihood differences show which tokens push the model toward each class. Holdout evaluation returns aggregate metrics without returning individual rows.

How WebMCP is used

The page registers nine tools with document.modelContext.registerTool:

Tool	Access	Job
get_training_state	Read	Model, queue, configuration, and metrics
inspect_uncertain_samples	Read	Entropy-ranked unlabeled tickets
inspect_model_diagnostics	Read	Features, confusion counts, calibration, and loss
predict_ticket	Read	Probabilities, entropy, and route-or-review decision
evaluate_model	Read	Aggregate holdout metrics
inspect_training_history	Read	Checkpoints and label provenance
queue_label_review	Write	A bounded request for human labels
train_confirmed_batch	Write	Incremental training on confirmed examples
propose_model_config	Write	A pending configuration proposal

Each definition has JSON Schema and runtime validation. queue_label_review has no label input. Samples move through unlabeled, awaiting-human, confirmed, and trained states, and training accepts only confirmed examples. Configuration changes require a separate human action.

Challenges

WebMCP browser support was the first practical problem. We verified all nine tools in Chrome 152 and built a deterministic replay for ordinary browsers. Replay events are labeled local replay so they cannot be confused with external calls.

The second problem was label provenance. Hiding a control in the interface would not prevent an agent tool from sending the same data. We removed labels from the review-request schema and made the training operation check sample state at runtime.

Evaluation tools return aggregate metrics, feature weights, and confusion counts while omitting individual holdout examples and answers.

We also corrected a misleading setting whose original name implied automatic training. It is now a review threshold. Predictions below it go to a person, and training remains limited to confirmed examples.

What we learned

Useful WebMCP tools matched domain operations instead of interface controls. queue_label_review gives the agent a clear capability and limit; a tool such as click_review_button would preserve the ambiguity of the page.

Human approval also needs a technical boundary. Label Loop keeps label assignment out of the agent tool set and checks provenance again when training starts. Since the model runs locally, every displayed probability, token weight, and metric can be reproduced without an account or API key.

Built With

• codex
• javascript
• vite
• webmcp