Evidence Workbench

Why we built it

A higher model score is only useful if the comparison is sound. While developing MAESTRO, we wanted a small tool that checks the evidence before anyone writes an improvement claim.

What it does

Evidence Workbench audits paired binary predictions. It checks subject separation and duplicate rows, then compares AUROC and Brier scores on the same test subjects. A paired, outcome-stratified bootstrap reports uncertainty around the AUROC difference. If the integrity check fails, it withholds the comparison. A downloadable report records the method and input fingerprint without exporting raw subject rows.

MCP interface

The official Python MCP SDK exposes synthetic_example and audit_predictions over Streamable HTTP. The browser demo runs both clean and leaky examples through real HTTP tool calls using protocol 2025-11-25. Structured responses limit a passing comparison to an exploratory claim and withhold claims when checks fail.

How to test

The private GitHub repository includes source, synthetic fixtures and setup instructions. Run the local browser server, then the MCP service in a second terminal. Open /agent-demo and run both toolchains. All 26 local tests pass.

Scope and next steps

This is a new supporting evaluation tool built during the hackathon, separate from the private MAESTRO model. Every result in the video is synthetic. It does not run MAESTRO inference, establish clinical validity or demonstrate a connection to Alexa. The next step is authenticated hosting and testing with an Alexa client. CSV checks cannot establish the absence of every form of leakage.

Build process

Python implements deterministic evaluation; HTML, CSS and JavaScript provide the browser interface. Codex assisted with implementation, testing and writing. Higgsfield edited real product captures into the caption-led demo. The source remains private for judging.