Vision-Based Research Copilot

Login Screen
The workflow
Report generated
Citation & Knowledge graph Image

Inspiration

Modern research is bottlenecked by the limitations of standard text-based AI. Traditional Retrieval-Augmented Generation (RAG) systems are fundamentally "blind" to the most critical parts of academic papers: complex methodology diagrams, intricate data tables, and spatial context. Furthermore, these systems cannot hunt for information themselves; they require a human to manually download PDFs and feed them into a text box. I wanted to build an agent that actually navigates the web to find research and uses eyes to understand it.

What it does

Vision-Based Research Copilot is an autonomous, multimodal research assistant that browses the web like a human to conduct comprehensive literature reviews. Operating primarily as a UI Navigator, the agent visually parses search engines, types queries, evaluates links, and navigates complex academic repositories to locate source PDFs. Once a paper is found, the agent executes a highly optimized Hybrid Multimodal Analysis Pipeline:

Targeted Visual Capture: Instead of inefficiently scrolling through 80 pages of text, the agent uses its vision capabilities to identify and capture high-signal visual elements (e.g., methodology diagrams, charts, tables, and reference lists).
In-Memory Deep Reading: Simultaneously, the agent extracts the full text of the document into memory, using a Map-Reduce approach to synthesize the dense academic text.
Unified Synthesis: It merges the visual insights (from the diagrams) with the textual synthesis to generate a rich, publication-ready research report and citation graph.

How I built it

The architecture relies heavily on the Agent Development Kit (ADK) and a robust Google Cloud backend to manage the complex state transitions of an autonomous browser agent.

Gemini 3 Flash (Computer Use): Powers the real-time UI navigation loop. It receives screen frames, interprets the spatial layout of academic databases, and executes keyboard/mouse actions to find papers.
Gemini 3.1-pro-preview & Gemini 2.5 Pro (Deep Reasoning): Handles the heavy lifting for the hierarchical Map-Reduce synthesis, processing massive 1M+ token contexts to analyze the extracted paper text and visual evidence.
Vertex AI: Serves as the core engine for all Gemini model routing and multimodal inference.
Google Cloud Storage (GCS): Acts as the immediate staging area for downloading PDFs and storing targeted screenshot evidence.
Cloud Run: Hosts the persistent backend WebSocket connections, streaming real-time JSON logs, visual frames, and citation graph updates directly to the frontend client.
Firestore: Manages the strict state-machine budgets (e.g., maximum vision calls, API rate-limit governors) to ensure the agent never gets trapped in infinite browsing loops and respects API quotas.

Challenges I ran into

The biggest hurdle was the "infinite scroll" trap. Initially, the vision agent would attempt to read 80-page PDFs by scrolling endlessly, which exhausted API rate limits and resulted in 429 Too Many Requests errors. I solved this by implementing a strict deterministic state machine. The system now enforces a "Screenshot Quota" (capturing 4-8 high-value visual areas) before handing the bulk text off to a backend in-memory extraction pipeline. This completely eliminated rate-limit crashes and made the agent incredibly fast and cost-predictable.

Accomplishments that we're proud of

I successfully bridged the gap between autonomous web navigation and deep document understanding. By creating a system that knows when to use vision (for UI navigation and diagrams) and when to use text extraction (for bulk reading), I built an agent that doesn't just read about research—it actively conducts it.

What I learned

I learned that relying purely on visual "click-and-scroll" loops for massive documents is highly inefficient and fragile. The key to building robust agentic systems is knowing how to strategically route multimodal tasks: using computer-use vision strictly for UI navigation and spatial understanding, while leveraging traditional backend pipelines and standard LLMs for bulk data extraction. Establishing deterministic guardrails and hard budgets is essential to move an AI agent from a fragile demo to a production-ready application.

What's next for Vision-Based Research Copilot

I plan to expand the agent's UI capabilities to navigate interactive data visualization dashboards (like Tableau or PowerBI) and integrate a continuous learning loop where the agent updates a persistent, personalized knowledge graph in Firestore over weeks of automated research.
Integration with Frontier AI Scientists: I envision this UI Navigator serving as the perceptual frontend for advanced reasoning models like Google DeepMind's Project Aletheia and AI Co-Scientist. While those models handle deep logical reasoning and hypothesis generation, the agent will act as their "eyes and hands"—navigating paywalled or complex journal UIs to fetch the papers and extracting the vital visual diagrams (charts, methodology graphs) they need to ground their research.