Agent Utility Index

A shared workspace for humans and AI agents to research, verify, and make evidence-backed decisions together.

Comment

Inspiration

Agent Utility Index did not start as a WebMCP project.

Before the challenge, I was benchmarking how AI agents retrieve information from the web. I tested retrieval providers across controlled datasets, comparing coverage, reliability, latency, evidence quality, and the cost of getting correct information.

Those experiments taught me something unexpected: better retrieval does not automatically create better agent decisions.

An agent can search more pages, use better retrieval infrastructure, and still leave the human with a basic problem:

What did it actually verify, what evidence supports each decision, and what happens when I change the requirements while it is working?

That became the inspiration for the WebMCP extension of Agent Utility Index.

Instead of building another research chatbot, I turned the webpage itself into a shared research workspace where:

Humans control the rules. Agents control the research.

And one rule is enforced by the system:

No source = no VERIFIED.

What it does

Agent Utility Index turns a webpage into structured shared state between a human and an AI browser agent.

A human creates a research objective and defines the must-have requirements.

The agent can then discover candidates, research them using its normal browsing capabilities, and write source-backed evidence into the board through WebMCP.

The board is not just displaying the agent's answer. It maintains the actual state of the research.

Every candidate is evaluated requirement by requirement. VERIFIED evidence requires a source URL. An unsupported attempt to mark a claim VERIFIED is rejected.

The human can also change requirements directly on the webpage while research is underway.

When that happens, the board creates a new revision and identifies the evidence checks that now need attention. Existing evidence that remains valid stays on the board instead of being discarded simply because the human changed one rule.

The result is a persistent research workspace that both sides can interact with instead of a decision existing only inside a chat conversation.

How I built it

The WebMCP version of Agent Utility Index exposes structured tools directly from the research page.

The core shared-board tools are:

  • create_research_board
  • get_research_board
  • add_research_candidate
  • update_candidate_evidence
  • get_missing_checks

The human controls the research requirements after board creation. The agent controls candidate discovery and evidence gathering. The state model uses stable IDs for requirements and candidates, board revisions, and revisioned evidence cells.This matters when human and agent actions overlap. Evidence writes are checked against the state they were based on so stale work cannot silently replace newer research.

Evidence is also validated at the tool boundary.

If an agent attempts to write: VERIFIED without attaching a source URL, the write is rejected.

With valid evidence attached, the same cell can be updated, the source becomes visible on the shared board, and the evidence revision advances.

get_missing_checks gives the agent a structured research queue containing unresolved evidence instead of requiring it to reconstruct the entire task from conversation history.

The page and agent therefore operate on the same research object through WebMCP.

Challenges I ran into

The hardest part was deciding what Agent Utility Index should do that a capable AI agent could not already do itself.

Modern agents can search the web, compare products, find sources, and make recommendations. Building another layer that simply performs those same searches would not add much value.

That pushed the project toward coordination rather than intelligence.

The agent is good at research. The webpage provides the persistent rules, evidence structure, revision history, and human control around that research.

Another challenge was concurrency.

A human can change a requirement while an agent is researching. An agent can also have multiple evidence operations underway. Board-level revision numbers alone were not enough, so the design evolved toward stable entities and evidence-cell revisions to prevent stale writes.

I also had to make the verification rule structural rather than relying on prompting.

Telling an agent to always provide sources is different from making the tool reject a VERIFIED write when the source is missing.

That distinction became one of the most important parts of the project.

Accomplishments that I'm proud of

I'm proud that the project evolved substantially instead of forcing the original Agent Utility Index concept into WebMCP.

The pre-challenge work was primarily about measuring retrieval systems. The WebMCP Challenge pushed the project into a human-agent collaboration problem.

The result is a working shared research surface where human actions and agent actions affect the same structured state.

I'm particularly proud of three things:

1. Verification is enforced rather than requested.

An agent cannot successfully write VERIFIED evidence without attaching a source.

2. Humans retain control of the evaluation criteria.

The agent can research candidates and evidence, but it cannot silently rewrite the rules it is being evaluated against.

3. Research is incremental.

Changing a requirement does not mean throwing away the entire research board. The system identifies unresolved evidence and exposes that work directly to the agent.

These are small primitives individually, but together they create a different interaction from receiving another generated comparison in chat.

What I learned

The biggest lesson came from the entire Agent Utility Index project, not only the hackathon.

At first I thought the hard problem was retrieval.

The benchmark work showed that retrieval matters. Different providers perform differently across workloads, and routing can reduce unnecessary retrieval work.

But once agents become capable researchers, another problem becomes important:

coordination between the person making the decision and the agent doing the work.

A chat transcript can describe a decision.

A shared stateful webpage can become part of the decision process itself.

WebMCP made that distinction much clearer.

I also learned that important constraints should be implemented as system behavior rather than prompt instructions whenever possible.

No source = no VERIFIED is much stronger when the tool refuses the write than when the prompt merely asks the model to behave that way.

What's next for Agent Utility Index

The original retrieval research and the WebMCP research board now point toward a larger idea.

Today, the agent can perform its own research and use Agent Utility Index as the shared decision workspace.

A future version could combine that interface with the retrieval intelligence explored in the earlier Agent Utility Index benchmarks, using specialized retrieval only when normal agent research cannot resolve an evidence check.

The board could also support:

  • larger research boards
  • evidence freshness and automatic re-verification
  • multiple human reviewers
  • richer source-quality scoring
  • reusable evaluation templates
  • exports of the final evidence-backed decision

Longer term, the same model could apply anywhere a browser agent is researching candidates against human-controlled requirements, including vendor selection, procurement, market research, due diligence, product comparisons, and service-provider evaluation.

The goal is not to make Agent Utility Index smarter than the agent.

The goal is to give capable agents a structured place to work with humans, where the rules remain human-controlled and the evidence remains inspectable.

Built With

Share this project:

Updates

Submission history