Inspiration
Our inspiration for the AI Browser Co-pilot came from a simple observation: the modern web is incredibly powerful, but it's also fragmented. As developers, researchers, and creators ourselves, we were constantly juggling a dozen different tools to do our work. We'd have one extension for performance analysis, another for scraping data, a separate web app for summarizing articles, and a constant back-and-forth between our browser and a separate AI chat interface.
We saw a clear need for a unified, intelligent assistant that could live directly in the browser and act as a true "co-pilot" for any task. We were inspired by the vision of an "all-in-one" super app that could not only answer questions but also see what we were seeing, understand our context across multiple tabs, and even help us debug complex technical issues. The announcement of Chrome's built-in AI APIs was the final catalyst. It gave us the missing piece: the ability to create a hybrid architecture that could be both incredibly powerful and deeply respectful of user privacy.
What it does
How we built it
We built the AI Browser Co-pilot on a modern, secure, and scalable on-device + Firebase cloud architecture, embracing a hybrid AI model from day one. Our project is composed of two primary components:
The Chrome Extension (Client) is the heart of the user experience, built with React, TypeScript, and Tailwind CSS and housed in Chrome's side Panel API for a seamless, integrated feel. The extension is more than just a UI; it's an orchestration layer. It uses Chrome's powerful APIs like chrome.debugger to capture detailed network data and getDisplayMedia to analyze on-screen content in real-time.
The Firebase Backend is our way to ensure that security and privacy are kept as our primary concern. We made the critical decision to never expose sensitive API keys on the client. Instead, we use the Vertex AI for Firebase SDK. When a user makes a request, the prompt is sent to our secure Firebase backend, which then uses its server-side credentials to call the Gemini 2.5 Pro model. This architecture gives us the power of a top-tier cloud model without compromising user security. The "magic" of our implementation is how these two components work together. For a feature like our "Conversational Network Analysis," the extension captures the raw data, the user asks a question, and our Firebase backend provides a deep, context-aware analysis, While our "Context Builder" feature within the extension scrapes data from multiple tabs, aggregates it, and sends it to the cloud for comprehensive, multi-document synthesis.
Challenges we ran into
Building a project of this complexity in a constrained browser environment came with a series of fascinating challenges:
Our biggest hurdle was the Cannot access contents of the page error. We learned the hard way that Chrome's Manifest V3 security model creates a strict separation of permissions between the service worker and other extension contexts. Our solution was to architect a "Request-in-Context" pattern, where the service worker opens a dedicated permissions tab, allowing the user to grant permissions directly to the service worker's context. It was a complex but architecturally correct solution that finally solved the problem.
When building the real-time video analysis, we discovered that calling getUserMedia from the side panel would cause the permission prompt to appear behind the main window or be instantly dismissed. This forced us to re-architect the feature to use a dedicated popup "capture window," which then streams frames back to the side panel. It was a great lesson in the nuances of browser security and user experience. For a long time, we were plagued by a auth/invalid-credential error. We meticulously checked every key and ID, but the error persisted. The breakthrough came when we realized that Firebase and Google Cloud have their own distinct configurations. The solution was to create the OAuth Client ID directly from the Firebase console's guided setup, which guaranteed the correct linkage between the two services.
Accomplishments that we're proud of
What we learned
This project was a deep dive into the present and future of web-based AI. We learned that the most powerful AI experiences are not "on-device vs. cloud," but "on-device and cloud." Our plan to integrate the built-in Summarizer and Prompt APIs as a fast, private "first line of defense," while reserving our powerful cloud model for complex tasks, is a testament to this. Our initial V1 prototype worked, but it was insecure and clunky. The migration to the V2 Firebase architecture was a huge undertaking, but it resulted in a product that is scalable, secure, and provides a far better user experience. We discovered that the quality of our AI's output was directly proportional to the quality of our prompts. For our conversational debugging feature, simply telling the AI to "act as a senior web performance engineer" and giving it clear instructions on how to interpret the data transformed its responses from generic statements to genuinely insightful, actionable reports. Building the AI Browser Co-pilot has been a challenging but incredibly rewarding journey. We've created a powerful tool that we, as developers and users, have always wanted, and we've laid the foundation for a truly intelligent future for the web browser.
What's next for AI Browser Co-pilot Super App
Release it in the wild and allow the world to experience the greatness that we have built, and continue to add features and iterate where we see gaps on the web browsing experience. More iterations within video and multimodality in general have massive opportunity within the chrome and on-device AI experience.
Log in or sign up for Devpost to join the conversation.