SYLVA: Resilient Web Automation via Visual Reasoning

A resilient web agent that navigates complex interfaces using vision-based reasoning with Gemini, bypassing traditional DOM-dependent automation limitations.

Comment

SYLVA: Resilient Web Automation via Visual Reasoning

Inspiration

Traditional web automation is trapped in the "DOM dependency" cycle. Even a minor UI update in a CSS class or nested structure often breaks existing scripts, leading to high maintenance costs. We asked: "Why can humans navigate any website intuitively while bots break at the slightest change?" SYLVA was built to break this cycle by treating the web as a visual canvas, using pure pixels as the ground truth.

What it does

SYLVA is an autonomous web navigator that "sees" websites like a human. It bypasses the DOM entirely, using high-resolution screenshots and Gemini 3 Flash's visual reasoning to navigate complex interfaces. It can handle dynamic layouts, SPAs, and cross-framework applications without needing any internal code identifiers.

How we built it

The project follows a Clean Architecture split between the Driver (Local Sentinel) and the Bridge (Reasoning Core):

  • Driver: Built with Python and Playwright, it handles asynchronous visual stability triggers and high-fidelity (JPEG 90) screenshot capture.
  • Bridge: Orchestrates the Gemini 3 Flash pipeline, implementing our proprietary Yield-Logic for value-based navigation.
  • Visual Grounding: We use a normalized coordinate system $[0, 1000]$ to map AI intent to physical pixels $P_{phys} = (\frac{x}{1000} \cdot W_{vp}, \frac{y}{1000} \cdot H_{vp})$.

Challenges we ran into

  • The "Coordinate Hell": In a vision-first system, being off by 5 pixels is a failure. We spent days debugging "AI hallucinations" where it would click just outside a button's hit-box, eventually implementing a strict Button Center Rule for mathematical grounding.
  • AI Diplomacy (Prompt Tuning): Gemini can be "lazy" or distracted by visual noise. We went through endless iterations to force it to verify its own actions and prevent aimless scrolling.
  • Latency Balancing: Achieving a sub-4s round-trip loop while maintaining high image quality required deep optimization of the capture and transmission pipeline.

Accomplishments that we're proud of

  • True Vision-First Navigation: We successfully implemented a system that is 100% independent of DOM structures.
  • Sub-4s Latency: Achieving near-real-time reasoning with high-fidelity visual data.
  • Self-Correction Logic: SYLVA can visually detect if a click failed and immediately pivot to a different coordinate or strategy.

What we learned

  • Pixels are the Ultimate Truth: While the DOM can be obfuscated, pixels don't lie. However, interpreting them requires "common sense" that must be baked into the AI's DNA (prompts).
  • The Symbiosis of Code and Prompt: Engineering an agent is 50% technical architecture and 50% "cajoling" the AI to follow strict logical constraints.
  • Resilience is Hard-Earned: Preparing for 100 ways an action can fail is what makes an agent truly "autonomous."

What's next for SYLVA: Resilient Web Automation via Visual Reasoning

Our vision for SYLVA includes:

  • Long-term Memory: Enabling the agent to learn a site's visual patterns over multiple sessions.
  • Multi-Agent Orchestration: Having multiple SYLVA instances collaborate on complex, multi-site workflows.
  • On-device Optimization: Reducing latency even further by moving more reasoning onto the edge.

Built With

Share this project:

Updates

Submission history