Inspiration
Most browser agents are fragile. They rely on DOM access, CSS selectors, or hardcoded API integrations — and break the moment a website updates. I wanted to build something different: an agent that navigates the web the way a human does — by looking at the screen.
I've spent months building Frankenstein AI, a cognitive architecture combining Hyperdimensional Computing, Active Inference, and Ebbinghaus memory. This hackathon was the perfect opportunity to connect it to Gemini 2.0 and give it eyes.
What it does
Gemini UI Navigator is a cognitive AI agent that sees and navigates any website through pure visual understanding.
You give it a task in natural language. It captures a screenshot, runs a three-system cognitive pipeline, assesses risk, executes actions, and learns from every interaction.
The Cognitive Cascade:
- System 0 — Deterministic. Known patterns execute instantly in under 100ms.
- System 1 — HDC Intuition. 4096-dimensional hypervector pattern matching (~500ms).
- System 2 — Gemini Deep. Full Gemini 2.0 multimodal visual analysis for unknown situations (5–15s).
Every action is filtered through Gut Feeling — a somatic marker risk detection layer that flags dangerous actions before execution.
First time filling a form: 11 seconds. Second time: under 100ms.
That's not optimization. That's the Ebbinghaus memory system learning.
How I built it
The backend is Python + FastAPI deployed on Google Cloud Run (europe-west1). Gemini 2.0 Flash handles all multimodal visual analysis via the Google GenAI SDK. A Playwright browser runs inside the container, capturing screenshots and executing actions. The Frankenstein cognitive architecture orchestrates the full pipeline — System 0 pattern matching, System 1 HDC intuition, System 2 Gemini deep analysis, and Gut Feeling risk assessment run on every single action.
Challenges
Getting real-time visual understanding fast enough for practical use was the core challenge. The three-system cascade solves this — Gemini only activates when faster systems can't handle the task. Cold start latency on Cloud Run required careful min-instance configuration for live demos.
What I learned
Gemini 2.0's multimodal capabilities are genuinely remarkable for UI understanding. It correctly identifies buttons, forms, dialogs, and navigation elements with high confidence even on sites it has never seen. Combined with a cognitive memory layer, the system gets measurably smarter with use.
What's next
Voice control via Gemini Live API — talk to the agent and it handles your computer. Real-time audio commands triggering the visual pipeline.
Log in or sign up for Devpost to join the conversation.