Why we built it
An ML experiment can report a higher score for the wrong reason. A changed candidate pool, evaluation split, preprocessing step, or metric definition can make two valid results unsuitable for direct comparison. A researcher may then spend more compute or change direction based on an apparent improvement that the evidence does not support.
Experiment trackers preserve runs, but researchers still have to reconstruct whether a comparison was fair, which sources support that conclusion, and what should happen next. A browser agent can help with this work, but its prose should never become the evidence or silently change the research record.
Research Audit Workbench turns that reconstruction into a reviewable collaboration between a researcher, deterministic audit rules, and a WebMCP-capable browser agent.
What it does
The application imports authenticated, owner-scoped research records from bounded prepared ZIP packages. Before anything becomes durable, it reconstructs both runs, inventories the source files, displays their metrics and provenance, and computes a canonical digest for the exact evidence package.
A deterministic comparability audit then evaluates:
- candidate pools
- evaluation splits
- preprocessing
- metric definitions
- missing evidence
- conflicting recorded and declared sources
- matched conditions
Recorded candidate manifests take priority over declared configuration. Critical mismatches block warnings, and matched checks are kept as separate findings with citations. Both reported metrics remain visible, and the audit avoids causal claims about why one score is higher. Overall confidence is bounded by the least-confident decision-relevant finding.
The introductory prepared example shows an obvious candidate-pool mismatch. The subtler example uses the same benchmark, candidate pool, split, preprocessing, and displayed metric label, but Run A uses micro-averaged nDCG@10 while Run B uses macro-averaged nDCG@10. The audit refuses to rank the runs as though those definitions were interchangeable.
After reviewing every finding and cited source, the researcher saves the complete audit package. A prominent human checkpoint asks the researcher to challenge or contextualize the interpretation without erasing the evidence. The application then generates a matched reevaluation plan from the entire audit, validates that every critical mismatch is resolved or explicitly retained as a named limitation, and binds approval to one exact plan version and SHA-256 digest.
Draft and approved JARVIS decision briefs can be downloaded as Markdown or JSON. Both formats come from the same canonical model. Drafts are visibly marked not approved; approved briefs are authoritative and preserve the comparison digest, runs, metrics, every finding and citation, researcher challenge, plan validation, exact plan digest, approval metadata, and the human-authority statement.
Why WebMCP is a strong fit
This is not a chatbot pasted beside a dashboard. The useful context already lives in the page: the selected runs, evidence sources, audit findings, current revision, plan, and approval state. WebMCP lets a browser agent work with that live state directly instead of asking the researcher to copy records into a prompt.
Active authenticated investigations expose seven tools:
get_current_comparisonrun_comparability_auditshow_finding_evidencestage_challenge_revisionstage_resolution_planreview_investigation_readinessget_decision_brief
The tools can inspect context, run deterministic analysis, focus cited evidence, and stage reversible previews. Approved-history pages expose only the three applicable read-only tools: comparison inspection, readiness review, and decision-brief retrieval.
No WebMCP tool can confirm an import, save an audit, persist a challenge, approve a plan, initiate a download, delete data, or execute an experiment. Those durable actions remain ordinary researcher controls. Registrations are tied to the current route and comparison digest and are aborted when either changes.
This creates a better human-agent workflow: the agent can do the tedious inspection and preparation while the researcher retains authority over every consequential transition.
How we implemented it
The Next.js App Router application renders the workbench, registers WebMCP tools through one feature-detected adapter, manages Supabase SSR authentication, performs owner-scoped database queries, and owns all human-confirmed durable Route Handlers.
An authenticated Next.js BFF loads trusted run snapshots, recomputes the selection digest, and forwards comparability audits to a stateless FastAPI service using the current access token. FastAPI parses bounded prepared packages, applies deterministic audit rules, emits evidence-backed findings, and validates reevaluation plans. It never receives database credentials.
Supabase provides authentication, PostgreSQL persistence, storage boundaries, row-level security, append-only investigation events, analysis revisions, and immutable approved plan versions. FastAPI's OpenAPI document generates checked-in TypeScript contracts.
The public disposable demo remains isolated from Supabase and uses versioned in-memory fixtures. In browsers without WebMCP support, the same research workflow remains available through manual controls.
Human authority and safety
Facts, severities, confidence, ordering, citations, and validation come from deterministic application code—not agent prose. The highest-severity finding becomes the primary challenge target, while the complete audit and all evidence remain in the durable revision.
The agent proposes. The application validates. The researcher decides.
That division is visible throughout the product: temporary imports before confirmation, reversible agent staging, a required researcher challenge, whole-audit plan validation, exact-digest approval, immutable approved history, and an explicit statement that no execution permission was granted.
Verification
The final source and deployment checks include:
- 25 passing Pytest tests
- Ruff
- 20 passing Vitest tests
- TypeScript type checking
- ESLint
- a successful Next.js production build
- deterministic OpenAPI and TypeScript contract generation
- 26 PostgreSQL pgTAP assertions for ownership, RLS, append-only history, and immutable approval
- all eight hosted Playwright journeys
The hosted browser coverage includes signup, both prepared imports, real multi-finding auditing, complete-package persistence, evidence inspection, the researcher challenge, matched plan editing, Markdown and JSON brief exports, exact approval, approved downloads, reload/history, the disposable demo, mobile overflow, keyboard focus, and the sticky JARVIS rail.
Current limits
Full-fidelity import supports the documented prepared research ZIP structure rather than arbitrary experiment folders. The product validates and approves a next-run plan but does not execute training or evaluation jobs. WebMCP remains experimental, so the application feature-detects support and preserves the complete manual path.
Built With
- fastapi
- next.js
- playwright
- postgresql
- pytest
- python
- react
- supabase
- typescript
- vercel
- vitest
- webmcp
Log in or sign up for Devpost to join the conversation.