AgentView

Inspiration

AI agents are being pointed at the web, but the web was built for human eyes, not agent reasoning. When an agent ingests a raw page, it drowns in navbars, cookie banners, legal footers, and decorative markup, burning tokens and making mistakes on content that has nothing to do with its task. Existing fixes like the Playwright accessibility snapshot are rule-based: they serialize everything the same way regardless of what the agent is trying to do, and they fall apart on messy, ARIA-poor pages. We wanted something smarter: a translator that reads a page the way an agent needs to, filtered by the agent's actual goal.

What it does

AgentView turns any webpage into exactly what an AI agent needs for its goal: relevant content, available actions, and nothing else. The contract is simple: (raw HTML, goal) -> AgentView JSON. It's goal-conditioned by design, so the same page with a different goal produces a different output: asking to "add the cheapest wireless mouse" returns the candidate products and their add-to-cart actions, while a different goal on the same page yields something else entirely. AgentView selects; the downstream agent decides. The output is a strict, capped JSON schema (click/type/select actions with deterministic element IDs) that an agent can execute directly through Playwright.

How we built it

We built a full distillation pipeline. First, an annotate -> pretrim input stage stamps deterministic data-av-ids on interactive elements and reduces the HTML to a dense, token-budgeted form: the same pipeline feeds every model so train==serve holds. We used Gemini as a teacher to generate goal-conditioned training examples, success-filtered them through a real agent-execution loop, then distilled that judgment into a small student model (Qwen3.5-4B + LoRA) fine-tuned and served on Freesolo Flash's GPU cloud. Around it we built a benchmark harness comparing conditions: prompted frontier models, our trained translator, and a real Playwright-MCP baseline, plus an MCP server, a MongoDB-backed results API, and a React dashboard to visualize the comparison. Every generated example must pass a validator that resolves selectors against both the trimmed DOM (what the model saw) and the raw DOM (what the executor drives).

Challenges we ran into

Keeping train and serve identical was the hard constraint: the training prompt has to be byte-for-byte what inference sends, so we funneled every prompt through one module. Validation was subtle: serialized HTML doesn't reflect typed text or checkbox state, so we had to write live input state back into attributes before serialization or multi-step labels came out corrupt. We also had to design the schema to be anti-reward-hacking array caps and selector rules that reject body-selectors, :nth-child, hallucinated text, and multi-match selectors because a distilled model will happily learn to game a loose contract. And token budget was a real cost lever: long context inflates VRAM and jumps the GPU class, so we ran a two-tier budget (a soft target for most pages, a hard gate for a deliberate long-page slice).

Accomplishments that we're proud of

We froze a clean, versioned contract and held it stable across the whole stack, schema, teacher prompt, validator, training, and dashboard all reference one source of truth. We proved out an end-to-end teacher→student distillation that produces a small, cheap model doing genuine goal-conditioned filtering, not just deterministic serialization. And we built an honest benchmark that holds the agent constant so we measure translation quality specifically, rather than conflating it with agent quality, including a real Playwright-MCP arm as a strong baseline.

What we learned

The judgment of what to keep is the whole product, that's the piece rule-based tools can't do and the piece worth distilling. We learned how much discipline train==serve demands, how easy it is for a model to reward-hack a loose schema, and how input-state fidelity (values, checked states) quietly determines whether multi-step traces are even valid. We also got a concrete feel for the cost curve of context length on GPU serving.

What's next for AgentView

Run the full teacher batch and scale the training set, then execute the planned ablations, pretrim-vs-raw-input now that raw pages can fit the larger context window. Push evaluation onto real public benchmarks (Mind2Web, WebArena-Lite, REAL) and messy real-world pages where the accessibility tree degrades most. Longer term: ship AgentView as a drop-in MCP tool so any agent framework can consume goal-conditioned pages, and expand the action taxonomy beyond the v1 click/type/select set.