Protocol Mirror

Did the trial publish what it registered? Protocol Mirror gives an AI agent WebMCP tools to trace the record, quote the evidence, and stage a proposal. A human makes the final call.

Comment

Inspiration

I have a passionate affection for medicine. I also love using AI in my day-to-day life. I use it to learn new topics or subjects, research problems and examples, assist me in making cool apps such as my AI trading bot, and turn my brain dump of ideas into a working prototype. All of these use cases are reasons why I love AI.

However, as I used AI more, one problem bothered me: AI, or an LLM, can give a fluent and confident answer even when the evidence behind it is incomplete or uncertain. In medicine and clinical research, we cannot afford to guess.

Clinical trials are a good example of this. A trial registry records what researchers planned to measure, while a publication reports what readers eventually see. Comparing them requires searching through multiple registry pages and carefully matching the exact wording. This can be exhausting for a person and often takes a long time to complete. An AI agent could make the investigation faster and more efficient, but giving it the final judgment would create another problem.

This problem inspired me to create Protocol Mirror:

“Can an AI agent do the difficult, intensive investigative work while keeping the final decision in the hands of a human?”

Protocol Mirror uses WebMCP to let an agent retrieve evidence, trace dated changes, quote exact source text, and propose a conclusion inside the webpage for the reviewer to see. The agent is useful, but it cannot approve or reject anything because that decision is left to the reviewer.

What it does

Protocol Mirror compares a clinical trial’s registered outcomes with what appears in its publication.

The default example uses the real ACTT-1 trial, connecting its ClinicalTrials.gov record and registration history with its PubMed report. Using WebMCP, the agent can:

  • Retrieve the live ClinicalTrials.gov record
  • Retrieve the live PubMed abstract
  • Examine the trial’s registration history
  • Find the outcomes that were registered first
  • Quote exact evidence with source locators
  • Stage a matched, omitted, introduced, or uncertain proposal
  • Request that a human review the proposal

The page initially shows seven tools. None of them can accept or reject a proposal. That decision is made by the human reviewer after reviewing the information.

Once the reviewer has made their decision, Protocol Mirror shows an eighth tool that lets the agent export a reviewed “receipt.” It includes only human-approved results, the evidence, source locations, and the audit history. If the reviewer undoes their decision, that tool is removed.

A session log also records what the agent has called and what the reviewer has decided. If the reviewer rejects something, the application requires a reason and allows the reviewer to add notes, which are then returned to the agent. This lets the agent revise the proposal instead of allowing the review to be the end of the conversation.

How I built it

I used Codex as an AI engineering collaborator throughout the entire project. My first prompts involved introducing Codex to the hackathon, providing it with the official rules, judging criteria, and WebMCP documentation, connecting the Devpost plugin to Codex, and researching common hackathon project patterns.

I first wanted to understand what WebMCP was and how it could change the structure of an experience instead of being used only as a decorative integration. After choosing clinical trials as the use case, I gave Codex a set of boundaries:

  • Use real ClinicalTrials.gov and PubMed records.
  • Preserve what the trial registered first.
  • Connect every proposal to exact quotations and source locations.
  • Treat external research text as untrusted.
  • Give the agent no accept or reject tool.
  • Make the human decision change the available tool surface.
  • Test with real cases and record model failures.
  • Keep benchmark results separate from clinical claims.

One prompt did not create this entire project. It took a lot of planning, analysis, and development for it to become what it is today.

Once Codex understood the challenge and had the necessary context, I used it to help create a plan, build the live-data adapters, model the evidence, implement the review workflow, improve the backend and frontend interface, examine security risks, and create the benchmarks. I reviewed each stage to find weaknesses before continuing.

Near the end, I used Claude Code and followed the same process. I gave it the necessary context and background, then asked it to critique my project and identify what I should fix. I used Claude for two reasons. First, I wanted a fresh set of eyes to inspect my codebase. Second, Fable 5.1 had been released, and its enhanced thinking, reasoning, and coding capabilities helped find and fix bugs.

I built the application with Next.js, React, TypeScript, GSAP, and WebMCP. Validated server routes retrieve public records from ClinicalTrials.gov and PubMed. The application converts them into stable outcomes and evidence spans with exact URLs and locators.

The tools are registered through document.modelContext.registerTool, with a fallback to navigator.modelContext. Pair-bound tools are registered again when the active trial changes. The receipt tool exists only while reviewed work exists.

Each tool has a closed input schema, bounded values, runtime validation, and an AbortSignal for cleanup. Read tools identify their output as read-only and untrusted. The application also checks outcome and evidence identifiers against the active case, preventing a proposal from citing an identifier that the reviewer cannot inspect.

I repeatedly asked Codex to reassess the project against the rubric at every stage, turning each weakness into the next engineering task to fix. The final checks covered WebMCP conformance, linting, TypeScript, the production build, 75 automated tests, and browser automation of the complete seven-to-eight-tool workflow.

Challenges we ran into

The hardest problem was enforcing human decision-making authority in the code. A title that says “human decision” proves nothing if the agent still has an action that lets it approve or reject a proposal. I removed that function entirely.

The reviewer decides through the application, and that decision changes which tool the browser registers. Getting dynamic registration, cleanup, undo behavior, and session restoration to work together took many revision sessions.

PubMed presented another problem. When I called its API, it returned abstract sections rather than a clean list of publication outcomes. I kept those sections labeled as abstract evidence instead of presenting them as something the source does not provide.

Testing also revealed browser-related behavior. WebMCP is new, and a local build has different origin requirements from the deployed application. I tested tool registration, live-source reads, proposals, reviewer feedback, receipt export, undo, reload, and case switching through the page’s actual WebMCP interface.

Another challenge was reporting failures honestly. The two local models made opposite errors. I reported the run as evidence for keeping a human reviewer, not as a general measurement of hallucination or clinical accuracy.

Accomplishments that I’m proud of

One thing I am proud of is that Protocol Mirror does not “fake” human control. The website and its guardrails prevent the agent from making the final decision.

The agent starts with seven tools that let it research the trials, quote evidence, and prepare a proposal for the reviewer. It has no tool or option that allows it to accept or reject its own work. Only after a person reviews the evidence and makes a decision is the eighth tool revealed, allowing the agent to export the reviewed results. If the decision is undone, that tool is removed.

These two guardrails prevent the agent from accepting or rejecting its own proposal and make the eighth tool available only after human approval. These guardrails do not prevent the agent from being wrong. They prevent an unreviewed proposal from becoming an approved result and keep the supporting evidence visible.

Another thing I am proud of is that the project is not just a demo product. It is a real-use-case project that can work accurately with real data and scenarios without failing.

I tested Protocol Mirror with 24 real trial-and-publication pairs. Every one of the 48 live record requests returned the requested source with usable evidence. I then gave that evidence to two small local AI models. One model reported changes too often, while the other missed most of the real changes it received. Their opposite mistakes showed why the evidence must remain visible and why a reviewer should make the final call instead of the agent.

I am also proud that the project gives reviewers a way to share their thoughts about why they rejected a proposal. They can provide a reason and write notes explaining their decision. This feedback is then returned to the agent, which can submit a revised proposal. A session log records the agent’s actions separately from the reviewer’s decisions.

Lastly, I am proud that I turned an idea combining medicine and AI into a working product. It uses live evidence from real cases rather than only demonstration data, includes a testable human checkpoint, and records how each conclusion was reached.

What I learned

The biggest thing I learned is that a confident AI answer is not the same as a reliable answer. I saw this when I tested two local AI models using the same kind of clinical evidence. One model reported changes too often, while the other missed most of the real changes it received. Both models could produce convincing answers with evidence, even though they were making opposite mistakes.

This changed how I thought about my product. Instead of trying to make the agent sound more certain about something, I focused on making its work easier for the reviewer to inspect and understand. The exact quotations, source locations, visible uncertainty, and review queue became more important than the agent’s final prediction.

I also learned that adding the title “Human review” is not enough. Human review needs to exist in the code, not only in the description or title.

If the agent had a tool that allowed it to approve or reject something, a message telling it to wait for the reviewer’s permission would only be a suggestion rather than an authoritative restriction. I solved this problem by removing the tool completely, leaving the final decision solely to the human.

We also added another guardrail. The final eighth tool becomes available only after the human reviewer makes a decision. If the reviewer has not made a decision, the eighth export tool is unavailable to everyone.

Finally, working with Codex taught me how important clear directions, correct prompting, verification, and complete context are when building with AI. Codex helped me significantly with research, creating a solid plan, executing and implementing features, testing, and revising the project at every stage.

However, Codex did not do all of the work. Just as Protocol Mirror requires a person to review the agent’s thinking and output, I still had to define the rules, inspect its work, identify weak ideas and problems, determine what the evidence supported, and decide what needed to be fixed.

What’s next for Protocol Mirror

Here is the plan for the future:

The first improvement I want to make is automatic matching between a clinical-trial registry and its publications. A reviewer could enter one trial identifier, and Protocol Mirror would find possible papers for review.

I also want to support more trial registries and full-text research papers. The current version uses ClinicalTrials.gov records and PubMed abstracts. Full-text sources would allow the agent and reviewer to inspect evidence that may not appear in an abstract.

The benchmark is another area I want to expand. The current test contains 24 real trial-and-publication pairs. A future version could collect and use a larger dataset reviewed by independent research experts. This would make it possible to study where different models fail without presenting the results as clinical validation.

I would also like to package the same human-authority design as a skill for other local models and LLMs, or apply it to other fields where an agent could be useful for gathering evidence but should not make the final decision.

The central rule would remain the same. I would remove the agent’s capability to make the final decision and include the additional guardrail of presenting certain tools only after the human reviewer has responded.

Built With

Share this project:

Updates

Submission history