GeminiNav

AI agent that sees and navigates any website using Gemini vision - your universal web autopilot on Google Cloud

Comment

Inspiration

Every day, millions of people perform repetitive tasks across websites - filling forms, searching for information, navigating complex UIs. What if an AI agent could just see the screen and do it for you?

Current browser automation tools require site-specific selectors, APIs, or DOM access. They break when websites update. GeminiNav takes a fundamentally different approach: it looks at the screen like a human would, understands what it sees using Gemini's multimodal vision, and takes action.

What it does

GeminiNav is a universal visual web automation agent. Give it a natural language instruction like "Search Google for climate change news and summarize the top result" and it will:

  1. Screenshot the current browser page
  2. Analyze the screenshot using Gemini 2.5 Flash vision
  3. Decide what action to take (click, type, scroll, navigate)
  4. Execute the action via Puppeteer
  5. Repeat until the task is complete

It works on ANY website with zero configuration. No selectors, no APIs, no site-specific code.

Key Features

  • Universal compatibility - works on any website without configuration
  • Natural language tasks - describe what you want in plain English
  • Visual step log - see screenshots and actions at every step
  • Smart termination - knows when a task is done (max 20 steps)
  • Concurrent sessions - multiple users can automate simultaneously

How we built it

Architecture:

  • Backend: Node.js + Express hosted on Google Cloud Run
  • Browser Engine: Puppeteer with headless Chromium
  • AI: Google GenAI SDK with Gemini 2.5 Flash (vision model)
  • Frontend: Clean web UI with real-time step visualization

The Agent Loop:

User instruction → Screenshot page → Send to Gemini Vision → 
Parse action (click/type/scroll/navigate) → Execute via Puppeteer → 
Screenshot again → Repeat until done

Each step produces a JSON response from Gemini specifying the action type, coordinates, and text. The agent validates and executes each action, then captures a new screenshot for the next iteration.

Challenges we ran into

  1. Screenshot-to-action accuracy - Gemini sometimes identifies elements slightly off-center. We added coordinate validation and retry logic.
  2. Knowing when to stop - The agent needs to recognize task completion. We implemented a "done" action type that Gemini can emit when the goal is achieved.
  3. Page load timing - Screenshots taken before page renders lead to incorrect actions. Added intelligent wait states between actions.
  4. Concurrent browser sessions - Each user needs an isolated browser context. Puppeteer's browser contexts solved this cleanly.

Accomplishments we're proud of

  • Zero-config universality - truly works on any website
  • Sub-3-second per step - fast enough to feel interactive
  • Clean architecture - the entire agent loop is ~100 lines of code
  • Google Cloud native - runs on Cloud Run with automatic scaling

What we learned

  • Gemini 2.5 Flash's vision capabilities are remarkably accurate for UI element identification
  • The screenshot-analyze-act loop is a powerful paradigm for visual automation
  • Keeping the agent loop simple and deterministic produces more reliable results than complex planning
  • Google Cloud Run + Puppeteer work surprisingly well together with the right Dockerfile setup

What's next for GeminiNav

  • Multi-step task planning - break complex tasks into subtasks
  • Memory across sessions - remember user preferences and login states
  • Voice control - combine with Gemini Live API for hands-free web navigation
  • Browser extension - run as a Chrome extension for even easier access
  • Enterprise workflows - automate repetitive business processes across SaaS tools

Built With

Share this project:

Updates

Submission history