WebWeaver

## Inspiration

Browsing the web on behalf of a user — booking flights, comparing prices, filling forms — requires understanding both the goal and the messy reality of dynamic web pages. We wanted to see if a team of specialized Gemini agents could handle the full loop: planning, executing, verifying, recovering from errors, and delivering a clear answer. No single model call can do that reliably, so we built a multi-agent pipeline where each agent owns one responsibility.

## What it does

WebWeaver transforms plain English goals into fully automated browser workflows using 6 specialized Gemini agents. A user types a goal like "Find the cheapest flight from Berlin to Istanbul on Google Flights," and the system takes over: a Planner decomposes the goal into atomic subtasks, a Navigator drives a real Chromium browser using Gemini Computer Use, and a Verifier checks screenshots and the accessibility tree to confirm each step succeeded. When things fail, an Error Recovery agent re-strategizes and retries. A Summarizer delivers the final answer. Everything streams in real time to a live dashboard.

## How we built it

We built on Google ADK (Agent Development Kit), using BaseAgent to create the orchestration framework. Each of the 6 agents has its own Gemini model, system prompt, and function-calling schema. The browser layer uses Playwright with headless Chromium, driven by Gemini Computer Use — screenshots go in, coordinate-based actions come out. We built a DOM compression system that extracts the accessibility tree via CDP and prunes it to interactive elements within 5 depth levels, cutting context by ~90%. The frontend is vanilla HTML/CSS/JS with WebSocket-driven updates. The whole stack is containerized with Docker and deployable to Google Cloud Run with a single script.

## Challenges we ran into

DOM compression was the biggest challenge — full page DOMs can exceed 100K tokens, making them unusable as model context. We iterated on pruning strategies until we found that extracting the accessibility tree and filtering to interactive elements within 5 levels gave agents enough to reason about the page. Timeout management was another pain point: without nested timeouts (45s per model call, 60s per CU step, 90s per sub-agent, 150s for Navigator), a single slow response would block the entire pipeline.

## Accomplishments that we're proud of

The multi-agent verification loop genuinely catches real failures. Single-shot browser automation breaks constantly on dynamic SPAs — dropdowns don't open, autocomplete menus overlay inputs, page transitions take unpredictable time. Our Verifier + Error Recovery loop detects these and retries with modified strategies. We're also proud of the real-time dashboard — every action, screenshot, DOM snapshot, and verification decision is visible as it happens, making the system fully transparent.

## What we learned

• Accessibility trees > raw HTML for LLM reasoning — the compressed [role] name = "value" format is more reliable than DOM markup
• Multi-agent verification loops catch failures that single-shot approaches miss entirely
• Event streaming architecture gives real-time observability for free when designed into the orchestrator from the start
• Timeout layering is essential — nested timeouts at every level prevent cascading hangs

## What's next for WebWeaver

• Voice input via Gemini Live API — speak goals naturally instead of typing
• Multi-tab support — handle workflows spanning multiple browser tabs
• Persistent task history — save and replay past automation runs
• Parallel subtask execution — run independent subtasks concurrently
• Custom agent pipelines — let users configure which agents run and adjust verification thresholds

Built With

• fastapi
• google-adk
• google-genai
• playwright