Traces

An AI agent can read your code but not your page, so it guesses why the button is dead. Traces gives it 17 WebMCP tools to search inside a recording, and to ask a human what it cannot see.

Comment

Inspiration

While building Xurel I hit a login form that would not submit. The email was valid, the password was valid, both fields were exactly what the standard asks for — and the button still did nothing.

So I did what everyone does now: I handed it to an AI agent. It read the code, guessed, was wrong, guessed again. It could see the source. It could not see the page. Every guess cost tokens and told us nothing about the one thing that mattered — what state the form was actually in at the instant the button refused. We fixed it in the end by hand, by looking.

Every engineer gets the six-word version of that bug from someone else: "checkout is broken, please fix it." No steps, no browser, no timestamp. It is the same problem one step removed — the state that would explain it is already gone by the time anyone reads the report. Which is why the demo app here ships with a pay button that stays grey.

The state that explains a dead button exists for one instant, in one browser, and nowhere else. The DOM at 28.157 seconds is not stored data — it is the result of a computation: replaying thousands of mutation events up to that instant. No backend can return it. Only a live replay engine can, and only while it is running.

That makes the tab running the replay the only place in the world where "at what millisecond did the province dropdown become empty?" can be answered. WebMCP is the primitive that opens that tab to an agent. Debugging Xurel, I did not know it existed.

What it does

Traces loads an rrweb recording in the browser and registers 17 WebMCP tools on document.modelContext. In the demo, a human drops six words into the task queue and an agent investigates:

  • read_network — one request at 11,599 ms succeeded, 200 OK, and came back array, 0 items. A request that fails is easy to find. This one didn't fail.
  • bisect — the agent sends a predicate and the page runs a binary search across the replay timeline: 10 probes, 36 ms, ±250 ms, landing on 28,157 ms. The playhead visibly jumps ten times while it runs, so a human watches the agent's search as motion rather than as a log line.
  • read_dom_at — 1,845 characters of HTML arrive as 16 lines, 557 characters, and one of them is the entire bug: select#province[name=province] [empty options: 0].
  • ask_human_visual — the agent stops. It cannot see the screen, it says so, and it asks a human to look. The human drags the playhead to the moment and picks an answer, which comes back as structured data rather than prose: { choice: "looked normal but empty", markedTimestamp: 28157 }.
  • propose_hypotheses — four ranked explanations, top one at 0.55 confidence, each citing evidence the human can click to highlight on the timeline. The human promotes one.
  • propose_report — a bug report whose nine reproduction steps were each checked against the real recorded event stream. Zero unverified. The human approves it.

Six words in. A root cause with timestamped evidence out.

The human is the eyes. The agent is the reasoner. That is the exact inverse of an agent taking a screenshot and guessing.

Why WebMCP, and not an API

  1. The answer does not exist until the page computes it. read_dom_at(28157) is not a lookup. The page has to replay the mutation stream to that instant and rebuild the DOM. A server holding the same recording file cannot answer without becoming a replay engine itself.
  2. bisect does not fetch — it programs the page. The agent hands over an algorithm to run over time, and the page runs it ten probes deep. There is no request/response shape for that, and no amount of clicking or DOM-scraping produces it either.
  3. A human can be a tool. execute() returns a Promise, so a tool is allowed to simply not resolve. That turns a person into something the agent calls when it reaches the edge of its own perception. An HTTP API cannot block on a human who is looking at your screen.
  4. The side effect is on human attention. seek moves the playhead; annotate leaves a marker a person sees. The agent isn't reading state — it is pointing at evidence and saying look here. That only works because the tool provider and the user interface are the same page.
  5. Nothing has to leave the machine. A recording is a full reconstruction of a page and everyone on it. With WebMCP it stays in the tab, and the agent receives only the small compressed slices it explicitly asks for.

The constraint that made the design better

WebMCP tool results are a content array, and today "text" is the only specified type — "image" is still an open question. We could not send a screenshot if we wanted to.

Designing around that produced the two things we are proudest of:

  • An agent-legible DOM representation with a hard budget of 60 lines and 1,200 characters, enforced by a test rather than by good intentions. Interactive and state-bearing nodes only, attributes from a whitelist, text truncated, layout wrappers discarded. It gives an agent things a screenshot never could: it can read aria-invalid, count option elements that were never rendered, and tell a genuinely disabled button from a merely grey one.
  • ask_human_visual, for questions that honestly require eyes. Did this dropdown look broken, or normal but empty? No amount of DOM state answers that.

A screenshot would have been the lazy answer and a worse one — thousands of tokens per call, and the agent still guessing at state it cannot read off pixels.

How we built it

Next.js and TypeScript, rrweb for record and replay, zustand for state, Tailwind for a dense instrument UI. No backend. Recordings are static JSON; replay, binary search, DOM compression and report validation all happen in the tab.

lib/replay, lib/dom and lib/bisect are pure functions with no React, no store, and no knowledge that WebMCP exists — which is why the search algorithm and the DOM compressor have real tests. 309 tests pass. Three suites (compress-dom, bisect, evaluatePredicate) were written first as specifications and stayed red until the implementations satisfied them.

Security was a design constraint, not a closing paragraph, because the premise of this challenge is handing a language model influence over a live page:

  • Nothing from the model is executed. Predicates are a closed set of seven structured shapes — never a string, never eval, never new Function. A test greps the source and fails the build if either appears, so it cannot regress quietly under deadline pressure.
  • Every response has a size budget, because a tool returning 800 KB has silently denied service to every later call in the conversation.
  • Personal data is minimised at the recorder, not at the reader: maskAllInputs is mandatory, values are truncated to 20 characters when a tool reads one, and network bodies are summarised (array, 0 items) rather than forwarded.
  • Every recording in the repo is synthetic, produced by bugbait/ — our own deliberately broken checkout, which ships alongside so anyone can make their own.

Challenges we ran into

  • The UI was the part we underestimated. The first version of Traces was genuinely bad to look at, and fixing it took more passes than any single feature. What finally worked was turning taste into constraints the build enforces: a type scale with a 13px floor for anything a person has to read — a compressed 1080p frame turns 10px body copy into grey texture — and a radius scale that stops at 6px, replaced rather than extended, so rounded-lg does not resolve and "no oversized rounded cards" is a build error instead of a line in a document. It is still not a beautiful app. It is a legible one, which for an instrument is the part that matters.
  • bisect was too fast to see. Ten probes is about 36 ms of machine work — zero frames, invisible. We animate the probe trace on the timeline for the human's benefit and display the real elapsedMs next to it, rather than pretending the search was slow.
  • A blocking tool can hang its host. A call that never returns is a broken tool, so each blocking tool returns { status: "pending", ticket } after 8 seconds and the agent collects the answer later with the same ticket. A blocking tool that returns "still thinking" is a conversation.
  • A tool can be correct and still unusable, because a model misreads its schema. The closed predicate set started as a security property and turned out to be a usability one too: models fill in a schema far more reliably than they compose an expression, so bisect tends to work on the first try.
  • rrweb's defaults quietly break replay-heavy work. Without checkoutEveryNms, rrweb emits exactly one full snapshot, so every seek replays from the beginning — and bisect seeks ten times by design. Nothing errors; the player just crawls.
  • Tools run outside React. execute() is called by the browser at arbitrary times with no hooks available, and it needs to read state, write state, and sometimes wait for a human. That requires imperative access from outside the render tree, which Context plus useReducer structurally cannot give you.

What we learned

The schema is the interface. We spent more time making tool arguments legible to a model than making them expressive, and every time we chose the narrower shape the agent got more reliable.

And a constraint we resented turned out to be the best thing that happened to the design. If WebMCP had let us return images, we would have shipped screenshots, spent thousands of tokens a call, and never written the tool where the agent admits what it cannot see.

What's next

registerDynamicTool is the one stub left in the codebase, and nothing calls it yet: promoting a hypothesis should mint a verify_hypothesis_1 tool specific to that finding, so the tool surface itself changes in response to what the agent discovered.

After that, the honest gaps. read_network cannot see navigator.sendBeacon, WebSockets or EventSource. There is no importer for non-rrweb recordings. And bisect assumes its predicate is monotonic in the window — a condition that flips repeatedly yields one transition, not all of them.

None of these are things we decided against. They are things we ran out of days for.

Built With

Share this project:

Updates