WebPilot

An autonomous, multi step live web agent for Alexa+. Built with MCP 2025-11-25 and a multi provider LLM brain, WebPilot navigates the real internet safely with cryptographic purchase approvals.

Comment

Inspiration

The promise of Alexa+ is an autonomous agent that can actually do things for you, not just answer questions. However, the current landscape of AI agents is limited by two massive bottlenecks:

  1. The API Trap: Relying on site specific APIs limits agents to only a fraction of the internet.
  2. The Trust Gap: Users do not trust autonomous AI to make live financial purchases without guaranteed, secure approval gates.

We built WebPilot to solve both. We wanted an Alexa+ agent that navigates the real, open internet (the DOM) just like a human, using existing logged-in browser sessions, while wrapping all actions in enterprise grade cryptographic security so the AI can never spend your money without explicit consent.

What it does

WebPilot is a multi-step, autonomous web execution engine for Alexa+.

  • Universal Action Loop: Instead of relying on hardcoded site rules, it visually parses any website’s DOM, mapping interactive elements (like "Add to Cart" or "Search") to execute tasks across e-commerce, ERPs and productivity tools.
  • CDP Session Reuse: WebPilot connects directly to a user's running Chrome browser via Chrome DevTools Protocol (CDP). This allows it to safely utilize the user's existing authenticated sessions (like Amazon or ChatGPT) without getting blocked by bot walls or CAPTCHAs.
  • Cryptographic HMAC Approval Gate: Before any destructive action or purchase, WebPilot pauses execution and generates an HMAC-SHA256 token containing the exact cart state and price. It pushes a Multimodal Card to the user. Only when the user manually approves the transaction is the token validated and executed, providing absolute protection against prompt injections and live price drift.

How we built it

  • The Architecture: We built a custom MCP 2025-11-25 (Streamable HTTP) server in FastAPI. This seamlessly integrates all three core MCP primitives (tools, resources, prompts) to interface with our Next.js simulated Alexa+ frontend.
  • The AI Brain: To ensure resilience, we designed a multi-provider routing engine. It seamlessly switches between a local Ollama LLM (for privacy) and the AWS Bedrock Converse API (qualifying for the AWS Builder mini-challenge), with a deterministic Symbolic Engine fallback.
  • The Browser Engine: Powered by Playwright, utilizing strict ecommerce heuristics and negative guardrails (e.g., dynamically avoiding "Delete" buttons while prioritizing aria-labels on checkout flows).

Challenges we ran into

  • E-Commerce DOM Complexity: Sites like Amazon use complex, deeply nested <input type="submit"> structures instead of standard buttons. Our agent initially struggled to differentiate between "Add to Cart" and nearby destructive actions. We solved this by implementing strict negative guardrails in our DOM scanner and applying priority multipliers for value/ARIA-label matches, validating the fix with a robust 170-test suite.
  • The Login/Bot-Wall Problem: Standard headless browsers get instantly blocked by Cloudflare or 2FA. We engineered a seamless fallback mechanism that hooks into the user's active Chrome debug port (9222), leveraging their real cookies, or gracefully degrades to an isolated local profile if the port is busy.

Accomplishments that we're proud of

  • Production-Ready Stability: Achieving 170 passing tests with 0 failures in our pytest suite before submission.
  • Security First: Successfully building the Untrusted Content Firewall and HMAC-SHA256 Approval Gate, proving that autonomous agents can be trusted in high-stakes retail environments.
  • True Protocol Compliance: Implementing a genuine, fully compliant MCP 2025-11-25 architecture from scratch.

What we learned

We learned the intricacies of the new Model Context Protocol and how powerful the Streamable HTTP transport layer is for real time AI agents. We also learned how to build deterministic safety guardrails around non-deterministic LLM outputs, proving that "Human-in-the-Loop" can be a seamless UI feature rather than a bottleneck.

What's next for WebPilot

  • Native Alexa+ Device Integration: Porting the Next.js simulated multimodal cards directly to physical Echo Show devices.
  • Mobile Web Navigation: Extending our DOM parsing heuristics to support mobile view layouts via Appium.
  • Federated Agent Swarms: Allowing WebPilot to hand off highly specific sub-tasks to specialized micro-agents within the AWS ecosystem.

Built With

  • ai
  • automation
  • aws-bedrock
  • boto3
  • chrome
  • fastapi
  • json-rpc
  • llm
  • modelcontextprotocol
  • next.js
  • ollama
  • playwright
  • pytest
  • python
  • react
  • web-scraping
Share this project:

Updates

Submission history