MarketBench

Inspiration\n\nBrowser agents can complete a task and still leave reviewers unable to answer basic questions: What actions did the agent take? Did the policy really improve the outcome? Can the result be reproduced? MarketBench makes those questions measurable.\n\n## What it does\n\nMarketBench is an interactive evaluation lab for browser-agent market policies. A visitor selects a scenario and seed, runs a 12-week simulation, and gets:\n\n- a clear business outcome and plain-language insight\n- the full WebMCP tool-call and state history\n- integrity hashes for trace and final state\n- a deterministic baseline comparison\n- three-seed aggregates with mean, standard deviation, and worst-seed results\n- downloadable evidence for audit and replay\n\n## How we built it\n\nThe app is a React and TypeScript web application deployed on Vercel. WebMCP exposes typed browser tools for configuring and running experiments. The evaluation engine uses versioned deterministic fixtures for three scenarios and three seeds, so baseline and signal-aware policy runs can be reproduced exactly. Each step records its input, tool call, output, state transition, and hash.\n\nThe comparison view aggregates the same fixed seed set rather than presenting a single lucky run. It visualizes distributions and reports mean, standard deviation, and worst seed. The repository includes tests for determinism, trace completeness, hashes, aggregate math, and export structure.\n\n## Challenges\n\nThe main challenge was making rigor understandable. A raw trace is useful to an engineer but intimidating to a first-time visitor. We rebuilt the experience around a three-step flow, visible progress, a human-readable takeaway, compact trace summaries, and expandable evidence. We also made the distinction between deterministic policy fixtures and model output explicit.\n\n## Accomplishments\n\n- Full trace, state, and history export with integrity hashes\n- Transparent baseline vs signal-aware comparisons\n- Three-seed aggregate metrics and charts\n- A first-run experience that explains the value before exposing technical detail\n- A live, reproducible demo with tested fixtures\n\n## What we learned\n\nAgent evaluation needs both observability and a decision-relevant metric. Traces explain how an outcome happened; controlled repeated runs show whether the policy actually helped. Neither alone is enough.\n\n## What is next\n\nNext, MarketBench can add imported real-world task traces, more scenario packs, signed evaluation bundles, and CI checks that compare agent-policy revisions before deployment.

Built With

• webmcp