WebPilot

A cryptographically secure autonomous web agent powered by NVIDIA Nemotron and Nebius AI Cloud.

Comment

WebPilot: Secure, Cost-Optimized Autonomous Web Automation

1. Inspiration

The internet is built for humans, not machines. Traditional web automation relies on brittle APIs, static DOM selectors, and backend endpoints that break the moment a website updates its UI.

We were inspired by a singular question:

What if an AI agent could navigate the web exactly like a human, securely bypassing bot-walls, while dynamically optimizing its own computing costs?

We built WebPilot to bridge the gap between deterministic software and probabilistic AI. With the rise of autonomous agents, the missing links have been trust and infrastructure scaling.

By leveraging the Nebius × NVIDIA ecosystem, we set out to build an agent that uses high-speed, lightweight models for everyday tasks and heavy-reasoning models only when the stakes are high—all while keeping the human in the loop through cryptographic security.

2. What It Does

WebPilot is a multi-step, autonomous, live web execution engine. Instead of relying on closed APIs, it visually parses live DOM elements to interact with websites.

Key Features

  • Universal Action Loop: Scans the live browser DOM, identifies interactable elements, and decides the next action.
  • Intelligent Model Cascade: Dynamically routes tasks between lightweight models for fast, low-cost DOM filtering and larger models, such as NVIDIA Nemotron, for deep spatial reasoning and complex checkout flows.
  • Cryptographic Approval Gate: Pauses execution before destructive or consequential actions, such as purchases. It presents a multimodal approval card to the user and requires explicit HMAC-SHA256 signature validation before proceeding.

3. How We Built It

We architected WebPilot as a Model Context Protocol (MCP) 2025-11-25 server, strictly separating tools, resources, and prompts.

AI Cloud Engine

We integrated the Nebius Token Factory to serve NVIDIA Nemotron models. We built a custom routing engine that evaluates the complexity of the current browser view and routes prompts to the appropriate model.

The Routing Logic

To optimize responsiveness and stretch available credits further, we implemented a dynamic cost-latency routing function.

The decision engine selects the optimal model $m$ by minimizing the following cost function:

$$ \text{Model}^{*} = \arg\min_{m} \left( \alpha L(m) + \beta C(m) \right) $$

subject to:

$$ R(m) \geq \tau_{\text{task}} $$

Where:

  • (L(m)) is the expected latency of model (m).
  • (C(m)) is the token cost of model (m).
  • (R(m)) is the reasoning capacity of model (m).
  • (\tau_{\text{task}}) is the complexity threshold of the current DOM.
  • (\alpha) and (\beta) are weighting coefficients that balance latency and cost.

Agentic Execution

We used Playwright combined with the Chrome DevTools Protocol (CDP) on port 9222. This allows the agent to attach to a user's existing authenticated browser session and interact with websites in that session, including situations where authentication or anti-bot challenges are present.

Security

Transactions are verified using a standard HMAC construction to prevent unauthorized agent purchases.

The HMAC-SHA256 construction is expressed as:

$$ \operatorname{HMAC}(K,m)

$$

H\left( (K \oplus opad) \parallel H\left( (K \oplus ipad) \parallel m \right) \right) $$

Where:

  • (K) is the secret key.
  • (m) is the message being authenticated.
  • (H) is the cryptographic hash function.
  • (\oplus) denotes bitwise XOR.
  • (\parallel) denotes concatenation.
  • (opad) and (ipad) are the outer and inner padding constants.

4. Challenges We Ran Into

Context Window Bloat

Modern e-commerce DOMs are massive. Feeding raw HTML from a search page into the model's context window introduced unnecessary latency.

We built a preprocessing script that strips non-semantic tags and uses a fast pre-filter to shrink the DOM payload by 85% before it reaches the Nemotron reasoning models.

CDP Port Contention

Relying on local CDP injection through --remote-debugging-port=9222 is prone to port conflicts caused by operating-system background processes.

We engineered a fallback mechanism that launches an isolated, persistent local Chromium profile if the primary connection times out.

Streaming Over HTTP

Maintaining long-running Server-Sent Events (SSE) for the MCP protocol led to timeouts during heavy page loads.

We implemented a custom two-second heartbeat chunk to keep the stream alive.

5. Accomplishments That We're Proud Of

The Dynamic Model Cascade

We successfully engineered a system that seamlessly hands off deep-reasoning tasks to NVIDIA Nemotron without losing DOM context or breaking the JSON tool-calling schema.

Enterprise-Grade Reliability

We built and passed a robust 170-test pytest regression suite. It ensures our DOM-parsing heuristics remain stable across complex, nested architectures.

The Trust Breakthrough

We demonstrated that our HMAC-SHA256 Cryptographic Approval Gate can intercept, freeze, and secure autonomous transactions in real time, keeping the user in control of consequential actions.

6. What We Learned

Model Cascading Is the Future

Forcing a massive model to perform simple element classification wastes computational resources.

Using Nebius Token Factory to seamlessly switch between models based on DOM complexity significantly reduced task completion time while balancing reasoning quality and cost.

Protocol Standardization

Building strictly to the MCP 2025-11-25 specification forced us to cleanly decouple Playwright automation logic from LLM reasoning logic, resulting in a more resilient and maintainable codebase.

7. What's Next for WebPilot?

Our immediate next step is introducing a Multi-Agent Swarm architecture.

We plan to utilize Nebius Serverless Jobs to spin up multiple parallel NVIDIA-powered workers that can execute concurrent research tasks across different browser tabs.

We also want to optimize our visual engine using Vision-Language Models (VLMs) to bypass HTML parsing entirely, relying purely on visual bounding boxes to bring secure web automation to every local machine.

WebPilot's vision: Secure, intelligent, and cost-efficient autonomous web automation that works with the web as humans experience it—while keeping users in control.

Built With

Share this project:

Updates

Submission history