Inference Chip Index

Inspiration

Inference-hardware comparisons are easy to get wrong. A leaderboard can silently mix workloads, scenarios, accuracy targets, units, or system-level and per-accelerator numbers. Traditional browser automation makes that worse because an agent has to infer meaning from controls and tables.

I wanted a safer collaboration model: the website should expose the exact research operations it supports, the agent should act on the same visible interface as the human, and every answer should remain traceable to immutable benchmark evidence.

What it does

Inference Chip Index turns official MLPerf Inference v6.0 results into workload-specific, source-linked comparisons. Its leaderboard now exposes four imperative WebMCP tools:

1. list_inference_slices discovers valid exact comparison boundaries.
2. configure_inference_leaderboard changes the filters visible to the human.
3. compare_inference_accelerators compares 2–8 accelerator families inside one exact slice and renders the result on the page.
4. clear_inference_comparison removes the shared comparison without resetting the leaderboard.

The central safety rule is simple: a tool cannot claim that one chip is universally fastest. Every comparison is constrained to the same release, Closed division, workload, scenario, accuracy target, metric, and unit. Results include immutable GitHub source URLs and SHA-256 hashes. If evidence is missing, the tool returns an explicit missing-evidence reason instead of inventing a number.

How I built it

The application uses Next.js App Router, TypeScript, React, and Cloudflare Workers through OpenNext. The normalized data is generated deterministically from the pinned MLPerf Inference v6.0 commit 4d3916ac9cf474b679cdfcf492d43a0559418ad1.

During the WebMCP Challenge submission period I added the browser-native collaboration layer with the imperative document.modelContext.registerTool() API. Pure validation and comparison logic lives in src/webmcp/leaderboard-tools.ts; registration and visible state updates live in the leaderboard React component. An AbortController unregisters the tools cleanly when the component unmounts.

The WebMCP layer reuses the same checked-in dataset and exact-slice invariants as the site. It does not introduce a parallel ranking implementation, fetch untrusted runtime data, invoke paid endpoints, or expose secrets. Tool input schemas reject unknown fields and validate all slice IDs, vendors, submitters, grouping modes, metric views, and accelerator slugs.

Challenge-period work

The reproducible data index and Lucid API existed before the event. The WebMCP extension was built specifically during the challenge window. Public commits 7b89ee5 through fd9f65f add the imperative tools, shared-state UI, tests, documentation, security header, MIT license, and production URL documentation. The public deployment includes that extension.

Challenges I ran into

• Converting a dense technical leaderboard into a small, predictable tool surface without duplicating ranking logic.
• Keeping agent actions visible to the human while preserving manual control of the same React state.
• Preventing incompatible benchmark dimensions from being compared even when an agent supplies plausible-looking inputs.
• Explaining absent results as evidence gaps rather than treating them as zero.
• Preserving immutable provenance in every structured response without overwhelming the visual interface.

Accomplishments that I am proud of

• One agent call can build a three-vendor comparison that immediately appears in the human's browser.
• Every returned row stays inside an exact benchmark slice and links back to immutable source evidence.
• The tool surface is read-only with respect to servers and payments, while still making useful visible changes locally.
• The app contains 145 verified results, 50 submitted systems, 13 normalized accelerators, and 9 exact slices.
• The implementation passes type checking, 21 automated tests, and a production build.

What I learned

WebMCP is most valuable when it exposes domain invariants, not when it merely turns every button into a tool. A good tool contract can make an unsafe action impossible, keep agent and human context synchronized, and communicate uncertainty as structured data.

I also learned that browser-native tools can be a better accessibility and audit surface than invisible automation: the human sees the filters change, sees the comparison render, and can verify the same immutable sources the agent used.

What's next

• Add WebMCP tools for methodology explanations and dataset-diff inspection.
• Let a human pin a baseline row and ask an agent for compatible deltas.
• Extend the importer to later MLPerf releases while preserving immutable snapshots and tombstones.
• Add signed dataset manifests so agents can verify provenance independently.
• Study how WebMCP tool descriptions affect agent selection and user trust in data-heavy workflows.

Repeatable demo

1. Open the live leaderboard in a WebMCP-capable browser.
2. Ask: “List the available exact inference comparison slices.”
3. Ask: “For the GPT-OSS 120B Offline slice, compare AMD MI355X, NVIDIA B200, and Intel Arc Pro B60 using official system results.”
4. Observe the comparison appear in the shared page with values, source links, and missing-evidence handling.
5. Ask the agent to clear the comparison or configure another visible slice.

Verification

npm ci
npm run type-check
npm test
npm run build

Verified locally: TypeScript passed, 21 tests passed, and the production build passed. The deployed leaderboard registered all four WebMCP tools, and a live three-vendor comparison returned exact-slice values with immutable sources.

Built With

• ai-agents
• cloudflare-workers
• mlperf
• next.js
• openai
• react
• typescript
• webmcp