Private AI Assistant

💡 Inspiration

We love the power of modern AI, but we hate the critical privacy trade-off.

Today, using an AI assistant means sending your browsing history, your selected text, and your private queries to a third-party server. We were inspired to build the "no-compromise" AI assistant, one that delivers the powerful, modern features we expect, but runs 100% locally on the user's machine.

Our pitch is simple: We've built a 100% private AI assistant for Chrome. It delivers multi-modal power, web browsing & tool-calling, all securely on your machine. Full capability, zero data leaks.

We wanted to prove that privacy and power are no longer mutually exclusive.

🛠️ How We Built It

Private AI Assistant is a Chrome Extension that directly leverages Google's new built-in Chrome Prompt AI API. This allows us to use a powerful, on-device large language model without a single byte of personal data ever leaving the user's computer.

Our tech stack is lean and privacy-first:

• Core Engine: Google's Chrome Prompt AI API for all on-device LLM inference.
• Architecture: A Chrome Extension built with a background service worker (background.js), a content script (content.js), and a Side Panel UI.
• Frontend: Simple, clean HTML, CSS, and JavaScript for the side panel chat interface.
• Tool-Calling Framework: This was our "secret sauce." We built a custom function-calling layer in the background service worker.
  1. The user's prompt is augmented with a "list" of available tools (e.g., browseWeb(url), getActivePageContent()).
  2. The local AI model decides if it needs to use a tool and formats its response as a JSON function call.
  3. Our service worker parses this JSON, executes the real function (like fetch()-ing a URL or reading the DOM), and gets a result.
  4. This result is fed back into the model, which then generates the final, tool-augmented answer for the user.

🧗 Challenges We Faced

1. Reliable Tool-Calling on a Local Model: This was our biggest challenge. Unlike massive cloud-based models (like GPT-4 or Gemini Advanced), local models are smaller and require far more precise prompt engineering. Getting the model to consistently output valid JSON for our function calls, and only when needed, took dozens of iterations on our system prompt.

2. Working with a Brand New API: The Prompt API is new/updated, so documentation is still evolving. We spent a significant amount of time experimenting to understand its capabilities, limitations, and best practices.

3. Context Window Management: We had to be very smart about what we sent to the model. We're balancing the user's query, the conversation history, and potentially large amounts of text from the active webpage. We had to build logic to summarize and truncate this context to avoid overwhelming the on-device model.

🎓 What We Learned

1. On-Device AI is Ready for Prime Time: We learned that local LLMs are no longer a "gimmick." They are fast, capable, and, with the right engineering, fully able to handle complex tasks like function-calling.

2. Prompting is Everything for Smaller Models: The quality of the system prompt is the single most important factor for success. A well-designed prompt can make a 7-billion-parameter local model feel as smart as a model 10x its size.

3. Extensions are the Perfect AI "Orchestrator": The Chrome Extension architecture is the ideal "glue" for an AI assistant. The service worker is the perfect place to manage the AI session and orchestrate tools, while content scripts and side panels provide the seamless integration into the user's workflow.

🚀 What's Next for Private AI Assistant

This hackathon was just the beginning. Our next steps are to:

• Expand our tool library (e.g., interacting with Google Calendar, managing tabs).
• Add multi-modal support by allowing users to pass audio to the AI.
• Build a simple UI to let users add their own custom tools and functions.

Built With

• javascript
• react