Nexagent

NexAgent — Your Goal, Our Agent. Autonomous Web Tasks, Simplified.

Comment

NexAgent — Autonomous AI Web Agent Platform

Inspiration

Today's web automation tools remain notoriously brittle. Traditional RPA and scraping scripts break whenever a CSS selector shifts, a layout updates, or an unexpected modal pops up. Meanwhile, modern websites are dynamic, stateful single-page applications packed with asynchronous hydration, infinite scrolls, and anti-bot obstacles.

We asked ourselves: Why can't an AI agent browse the web the way a human does? A human doesn't parse raw HTML strings or hardcode XPath selectors; humans rely on visual cues, contextual hierarchy, and iterative trial-and-error.

We were inspired to build NexAgent: a truly website-agnostic autonomous web agent powered by advanced Vision-Language Models (VLMs) and Playwright. Our goal was to create an intelligent system capable of decomposing high-level user intents, perceiving dynamic browser viewports, adapting to unpredictable web interfaces, and knowing when to ask for human assistance instead of failing silently.

What It Does

NexAgent transforms natural-language objectives into deterministic, multi-step browser execution workflows across any web application without pre-configured integrations.

Key Capabilities:

Autonomous Multi-Step Goal Navigation: Translates open-ended prompts (e.g., "Find the top 5 mechanical keyboards on Amazon, compare customer ratings, and extract results into a structured table") into sequential, goal-directed browser actions.

Intelligent Dynamic Form Filling: Automatically comprehends complex forms (Google Forms, enterprise portals, multi-stage sign-ups), selecting checkboxes, radio groups, file pickers, and input fields with human-like typing cadences.

Multimodal Visual & DOM Grounding: Fuses high-resolution viewport snapshots with structured DOM accessibility trees, enabling surgical click/type accuracy via Vision LLMs.

Human-in-the-Loop (HITL) Sentinel: Detects bot challenges (reCAPTCHA, Cloudflare Turnstile, hCaptcha, 2FA OTPs). When detected, it pauses execution, alerts the user via an Action Needed banner, and offers an Interactive Takeover Canvas. Once resolved by the human, the agent automatically detects state progression and resumes autonomous execution.

Real-Time Ultra-Low Latency Screencast: Streams browser canvas frames over WebSockets at up to 30+ FPS with interactive human mouse clicks, scrolls, and keystroke pass-through.

Enterprise-Grade State Persistence: Tracks all task histories, step-by-step reasoning logs, DOM snapshots, and structured JSON outputs backed by PostgreSQL and Redis.

How We Built It

NexAgent is built on a high-throughput, decoupled microservice architecture:

┌────────────────────────────────────────────────────────┐
│             React 18 Frontend + Interactive Canvas      │
│        (Vite, TailwindCSS, WebSocket Stream Viewer)     │
└───────────────────────────┬────────────────────────────┘
                            │ HTTP API & WebSockets (/ws)
                            ▼
┌────────────────────────────────────────────────────────┐
│            FastAPI Asynchronous Backend Engine         │
│  ┌───────────────────────┐   ┌───────────────────────┐  │
│  │ Vision Planner & LLM  │   │ Headless Chromium Core│  │
│  │ (Nemotron-3.5-SuperVL)│   │ (Playwright Async API)│  │
│  └───────────┬───────────┘   └───────────┬───────────┘  │
└──────────────┼───────────────────────────┼──────────────┘
               ▼                           ▼
┌──────────────────────────────┐ ┌──────────────────────────────┐
│        PostgreSQL 16         │ │           Redis 7            │
│   (Relational Persistence)   │ │  (Pub/Sub & Low Latency Ephem)│
└──────────────────────────────┘ └──────────────────────────────┘

1. Vision & Reasoning Engine (Nebius AI Token Factory)

We leveraged the NVIDIA Nemotron-3.5-Super-VL multimodal vision model hosted on Nebius AI Token Factory. Unlike text-only models that struggle with dynamic SPAs, Nemotron VL processes the visual page canvas along with extracted interactive bounding coordinates.

2. Mathematical Formalization of Agent Decision-Making

We model the browsing session as a partially observable Markov decision process (POMDP):

$$ S_t = \langle I_t, \mathcal{T}{\text{DOM}}, \mathcal{H}{1:t-1} \rangle $$

Where:

  • (I_t \in \mathbb{R}^{H \times W \times 3}) is the viewport screenshot at step (t).
  • (\mathcal{T}_{\text{DOM}}) is the filtered Accessibility Tree containing interactive elements.
  • (\mathcal{H}_{1:t-1}) represents the action-history sequence up to step (t-1).

The policy network selects an optimal action (a_t^*) maximizing goal alignment:

$$ a_t^* = \arg\max_{a \in \mathcal{A}} \mathbb{P}\left(a \mid \mathcal{G}, S_t\right) $$

Where (\mathcal{G}) is the user's objective and (\mathcal{A}) is the action space:

$$ \mathcal{A} = { \text{click}(x,y), \text{type}(\text{selector},\text{text}), \text{scroll}(\Delta y), \text{navigate}(\text{url}), \text{extract}, \text{finish} } $$

3. State Fingerprinting & Loop Detection

To prevent recursive loops (e.g., clicking the same element repeatedly), we compute a cryptographically deterministic SHA-256 state fingerprint (\mathcal{F}(S_t)):

$$ \mathcal{F}(S_t) = \operatorname{SHA256}\Big( \text{URL} \parallel \text{Title} \parallel \text{InteractiveElementsHash} \parallel \text{DOMTextSummary} \Big) $$

If (\mathcal{F}(S_t) \equiv \mathcal{F}(S_{t-1})) after an action intended to cause mutation, our self-healing recovery module activates fallback heuristics or escalates to human intervention.

4. Interactive Canvas & Screencast Optimization

We developed an ultra-efficient binary WebSocket pipeline:

  • Playwright Chromium captures JPEG viewports at variable quality levels.
  • Downsampled stream frames are transmitted over /ws/live to an HTML5 <canvas> element.
  • Normalized mouse events ((x_{\text{canvas}} / W_{\text{canvas}}, y_{\text{canvas}} / H_{\text{canvas}})) are mapped back to real browser viewport coordinates for zero-latency manual interactions.

Challenges We Faced

Anti-Bot Defenses & Dynamic CAPTCHAs

Challenge: Cloudflare Turnstile and Google reCAPTCHA v2/v3 break autonomous automation loops.

Solution: Rather than employing fragile circumvention tricks, we engineered the Human-in-the-Loop Sentinel. It listens for iframe injection and known bot-check signatures:

$$ \mathbb{P}(\text{SentinelTrigger}) = \mathbb{I}(\text{CAPTCHA_DOM_match}) \lor \mathbb{I}(\text{Turnstile_iframe}) $$

When triggered, the session freezes autonomous actions and surfaces a live interactive control box to the user. Once the challenge passes, autonomy resumes seamlessly.

Screencast Bandwidth & Browser Lockups

Challenge: Continuous frame capture in Chromium causes significant event-loop starvation when concurrent LLM inference is running.

Solution: Implemented adaptive frame pacing and dirty-rectangle viewport checks. Frame streaming automatically throttles during heavy network loads and bursts during user mouse interaction.

VLM Token Hallucination on Dense Webpages

Challenge: Webpages with thousands of DOM nodes cause context overflow or spatial hallucinations.

Solution: We built a DOM accessibility tree extractor that prunes hidden nodes, ads, and unclickable metadata, passing only salient interactable element descriptors paired with visual screenshots.

Accomplishments That We're Proud Of

  • Truly Zero-Selector Automation: NexAgent can navigate completely unfamiliar websites without a single pre-written XPath or CSS selector.
  • Bimodal Human-AI Collaboration: The seamless handoff between AI autonomy and live human takeover creates an agent that never gets permanently stuck.
  • High-Performance Streaming: Sub-100ms visual interaction loop between the backend Playwright instance and the React frontend.
  • One-Command Docker Deployment: Complete orchestration of PostgreSQL 16, Redis 7, FastAPI, and Vite/Nginx in a single command (docker compose up).

What We Learned

Vision + DOM is strictly superior to pure DOM or pure Vision: Visual grounding alone struggles with obscured or off-screen elements; DOM trees alone lack spatial understanding. Combining both bridges the grounding gap.

Failure-mode handling defines agent utility: An agent that reaches an impasse and asks for human assistance with full context is vastly more practical than an agent that enters hallucination loops.

Deterministic state validation is mandatory: LLM planners must be constrained with state transition checks and goal verifiers to avoid premature declaration of mission success.

Built With

Share this project:

Updates

Submission history