XtraDevPilot

Give AI coding agents eyes, hands, and memory — enabling them to understand your codebase, interact with real applications, debug issues, and remember engineering decisions across sessions.

Comment

Inspiration

Traditional headless automation tools (Puppeteer, Selenium, raw Playwright) fail when dealing with modern authentication, Cloudflare bot challenges, CAPTCHAs, SSO sessions, and complex React/Vue state. We wanted to empower AI agents and developers to interact directly with their real, authenticated Google Chrome browser where all logins, cookies, and extensions already live.

What it does

Xtra DevPilot seamlessly connects Google Antigravity, Claude Desktop, Cursor, and autonomous AI agents directly to your real, authenticated Google Chrome browser via the Model Context Protocol (MCP) and a sub-12ms local WebSocket bridge (ws://127.0.0.1:42819).

Key features include:

  • Token-Optimized DOM Extraction: get_clean_dom_snapshot strips bloated SVGs, script tags, and styles to save up to 94% LLM context tokens.
  • Turbocharged Form Automation: batch_fill_form fills 20+ inputs in a single rapid roundtrip (<200ms) with full React/Vue state compatibility.
  • Smart Comboboxes: smart_select_combobox automates complex search-and-select dropdowns (Workday, Greenhouse, ARIA comboboxes).
  • Direct File Uploads: upload_file attaches local files directly to file inputs or drag-and-drop zones.
  • Live State & Store Inspection: execute_script evaluates JS in the page's MAIN execution context (Redux, Zustand, Pinia, React Fiber).
  • Instant Visual Debugging: toggle_layout_debug_mode renders high-contrast layout outlines; inject_css hot-patches CSS without reloads.
  • Deterministic QA & Test Synthesis: assert_element_state for pass/fail checks and record_user_flow to auto-generate Playwright test scripts.

How we built it

  • Chrome Extension (Manifest V3): Utilizes background service workers, content scripts in ISOLATED world, and injected scripts in MAIN execution context.
  • WebSocket Bridge Server: Ultra-fast Node.js bridge server broadcasting bidirectional JSON-RPC messages on ws://127.0.0.1:42819.
  • TypeScript SDK & CLI: High-level SDK and CLI tools for custom script execution and testing suite integrations.

Challenges we ran into

Building a reliable bridge across Manifest V3 isolated script contexts and main world execution contexts without breaking Chrome security policies required custom postMessage routing and DOM event dispatches. Ensuring token efficiency for large dynamic DOM trees led us to architect specialized DOM sanitizer algorithms.

Accomplishments that we're proud of

  • Verified 33 live-verified tools with 100% pass rates against active authenticated Chrome sessions.
  • Reduced LLM token consumption by 94% through DOM cleaning algorithms.
  • Created seamless 1-click Playwright test synthesis directly from user web sessions.

What we learned

Optimizing browser-to-AI context passing requires careful DOM tree pruning, React event synthetic dispatching, and asynchronous WebSocket handshake resilience.

What's next for XtraDevPilot

  • Expanding multi-browser support (Firefox, Safari, Brave).
  • Enhanced visual multi-modal spatial grounding and element identification.
  • Community plugin registry for custom domain-specific automation recipes.

Built With

Share this project:

Updates

Submission history