Experiment Guardian: MCP-Powered ML Reproducibility

Inspiration

ML experiment evidence often lives in different places: code in GitLab, run parameters in MLflow, and decisions in meeting notes. When they disagree, reviewers have to reconstruct what happened. Experiment Guardian brings those records into one reproducibility review.

What it does

A researcher supplies a GitLab merge request URL, an MLflow run ID, and a JSON training-file path. Guardian reads the file at the merge request’s pinned commit, retrieves the run, and saves an evidence snapshot. Deterministic rules flag conflicting or missing values such as the seed, learning rate, dataset version, and evaluation split.

The researcher can discuss the same review in an Alexa+ web simulation that calls Guardian’s self-hosted MCP server over Streamable HTTP. It asks focused questions and accepts notes or documents. Proposed corrections require human approval before Guardian changes the record and reruns the audit. The final report keeps unresolved findings visible.

How we built it

We built the frontend with React, TypeScript, Vite, and Tailwind CSS. An Express backend uses Zod for validation, SQLite for evidence and decision history, and GitLab and MLflow APIs for source data. The browser uses the official MCP TypeScript SDK to display actual tool calls. Local checks negotiated MCP protocol version 2025-11-25.

Challenges we ran into

We had to keep a researcher’s explanation separate from verified run evidence. We also needed to distinguish an unavailable artifact listing from proof that artifacts were missing. GitLab identity verification blocked hosted CI for the demo fork. Account restrictions blocked live Amazon Bedrock and Transcribe calls, so the recorded linked-run demo does not depend on them. Its Alexa+ experience is a web simulation, not the official Alexa+ client or an autonomous LLM.

Accomplishments that we're proud of

The Iris demo links a real GitLab merge request and MLflow run. Guardian catches a deliberately incorrect draft seed of 7 against the run’s recorded seed of 42, then guides an approved correction. The same review can import a note about the test split and finish with an audit report. Local verification exercised all 19 advertised MCP tools.

What we learned

Conversation is valuable for preserving reasoning, but it should not silently overwrite experiment evidence. Explicit approval, pinned source links, and honest “unverified” findings make a review easier to trust.

What's next for Experiment Guardian

We plan to add durable team storage, individual permissions, broader training-config support, and stronger artifact verification. We also want to validate the advisory GitLab CI job in a project that can run pipelines and enable Bedrock extraction when model access is available.

Built With

• alexa+
• docker
• express.js
• gitlab-api
• mcp
• mlflow
• node.js
• react
• render
• sqlite
• streamable-http
• tailwind-css
• typescript
• vite
• vitest
• zod